Anyone who has worked with real-world data knows that the hardest part is rarely the model—it’s the data itself. Missing values, inconsistent formats, duplicates and hidden errors can silently undermine even the most sophisticated AI systems. In practice, a significant portion of time is spent not on building models, but on cleaning and preparing data.
Why data cleaning is difficult
Nonetheless, data cleaning is no trivial task. We can separate it into two sub-tasks:
- Error Detection, and
- Error Correction.
In the former, we discover errors in values, while in the latter we attempt to replace them with the correct ones. Errors can take many forms, such as domain discrepancies (e.g., outliers), misspellings, or inconsistencies across columns, such as for example, two columns representing the start and end of a process sharing the same timestamp. After identifying these issues, the appropriate correction depends on the type of problem: can we use existing values to infer a solution (e.g., the average of a column), or should we rely on predefined rules?
These questions directly affect both the effectiveness and efficiency of data cleaning methods. As a result, domain experts are often involved in the process (Human-in-the-Loop), since automated approaches typically require manual validation at later stages.
Within the AI-DAPT project, data cleaning is a key component of building trustworthy and adaptive AI systems. The platform aims to support end-to-end AI pipelines, where high-quality data is essential for reliable outcomes. In this context, our tool focuses on improving the data preparation stage by combining automated methods with user-driven validation.
The Data Cleaning Engine
The Data Cleaning Engine is a component of the AI-DAPT pipeline (see Figure 1), designed to detect and repair data quality issues during the Data Design for AI phase. It supports two main data types, Time Series and Tabular Data, and focuses on improving data readiness for downstream AI tasks.
At its core, the engine addresses two fundamental tasks in data cleaning. First, Anomaly (Error) Detection, which identifies erroneous or anomalous values and replaces them with NaN, preparing the data for subsequent correction or imputation steps. Second, Data Imputation, which fills missing (NaN) values in a dataset and returns a DataFrame of the same shape with completed values.
What distinguishes this component is its modular design, where detection and correction are treated as separate but composable operations. This allows different strategies and methods to be integrated depending on the data characteristics and application needs, rather than relying on a single fixed pipeline.
👉 Prefer to see it immediately in action? Jump directly to the demo video below.
Supported capabilities and methods
Currently, the Data Cleaning Engine supports Time Series data, while Tabular data support is under active development. For Time Series, it provides a set of interchangeable methods for both anomaly detection and data imputation, allowing flexible configuration depending on the use case.
The table below summarizes the currently supported methods for each task and data type.
| Task | Generic | Tabular | TimeSeries |
| Data Imputation | backward_forward_fill, mean_fill, median_fill, mode_fill, linear_interp_fill, polynomial_interp_fill, spline_interp_fill, time_fill, nearest_fill, pad_fill | saits, brits, usgan, mrnn, gpvae, timesnet, nonstationary_t ransformer, autoformer, units, moment, e2e_ppg | |
| Anomaly Detection | matrix_profile, zscore_outlier_detection, iqr_outlier_detection, isolation_forest, angle_base_outlier_detection, clustering_based_local_outlier, connectivity_based_local_outli er, histogram_based_outlier_detect ion, k_nearest_neighbors_detector, local_outlier_factor, one_class_svm_detector, principal_component_analysis, minimum_covariance_determinant, subspace_outlier_detection, stochastic_outlier_selection | moment, e2e_ppg |
Human-in-the-loop & UI
The component supports a human-in-the-loop approach, allowing users to configure the operations to be performed and review the results. It is integrated into the AI-DAPT platform as a DAVE Operator, enabling users to build data pipelines within DAVE and seamlessly include the data cleaning task. Through the dedicated UI, users can configure methods, execute operations and inspect results at each step of the process.
A typical workflow is illustrated in Figure 2. Data is first loaded, followed by triggering anomaly detection. The user then configures and runs the detection step, inspects the results and proceeds to the imputation phase (see Figure 3). After configuring and executing imputation, the corrected dataset can be inspected and passed to subsequent pipeline tasks.
Watch It in Action 🎥
The following demo illustrates the end-to-end workflow of the Data Cleaning Engine on a simple dataset example with two numerical columns. It showcases the two main stages of the pipeline: anomaly detection, where erroneous values are identified and marked, followed by data imputation, where missing or corrupted values are filled to restore the dataset. The example demonstrates how users can interactively configure and execute both steps through the UI.
This demo was captured and narrated by our project partner, Alexandro Zeakis.
Current status & roadmap
In the current alpha version of the project, we focus on time series data, with the initial set of detection and imputation methods already integrated into the Data Cleaning Engine. In the next phase, we will extend support to tabular data, broadening the applicability of the system to a wider range of use cases.
In the beta version, we plan to incorporate Large Language Models into the cleaning pipeline, enabling more flexible and context-aware decision-making. At the same time, the role of the domain expert will remain central, ensuring that automated suggestions are validated and aligned with real-world requirements.
Closing
Data cleaning remains a critical step in building reliable AI systems and the Data Cleaning Engine aims to make this process more structured, interactive and adaptable. By combining automated methods with human expertise, it supports more robust and trustworthy data preparation within the AI-DAPT ecosystem.