Unified Frameworks for Multi-Error Data Cleaning

20 concepts13 papers1 proposal

Data cleaning has traditionally been treated as a series of isolated, specialized tasks. If a dataset contained spelling mistakes, practitioners applied an edit-distance spellchecker. If it violated relational constraints, they executed a functional dependency validator. However, real-world data rarely confines its flaws to a single category. Instead, dirty data presents a complex, overlapping mixture of schema-level violations, formatting inconsistencies, missing values, and semantic contradictions.

Early foundational work by Rundensteiner et al. in their Letter from the Special Issue Editor established a multi-dimensional taxonomy recognizing that data errors span both schema and instance levels across single or multiple sources. Despite this early awareness, data cleaning tools remained heavily siloed.

In practice, this fragmentation creates a fundamental engineering bottleneck. As demonstrated empirically by Abedjan et al. in Detecting Data Errors: Where are we and what needs to be done?, no single data cleaning tool is universally effective across all error types and datasets. This forces practitioners into a manual, trial-and-error process of stitching together disparate tools—a problem highlighted by Mahdavi et al. in Towards Automated Data Cleaning Workflows. To move beyond this "one-size-fits-all" failure, a modern research direction focuses on unified data cleaning frameworks capable of detecting and repairing heterogeneous, interacting errors simultaneously.

The Hidden Interplay of Compound Errors

Why can we not simply chain specialized cleaning tools together in a sequential pipeline? The reason lies in the concept of compound errors: different types of data corruption actively interact with and mask one another.

Consider a table where a single row contains both a typo in a city name ("New Yrok") and a violation of a functional dependency rule linking zip codes to cities. If an isolated spellchecker runs first, it operates blindly without understanding the table's structural rules; it might suggest an incorrect semantic repair based purely on string distance. Conversely, if a rule-based constraint validator runs first, it will flag the row as invalid but lack the statistical context to know that a simple typo caused the structural alarm.

This destructive interplay manifests across various domains:

  • Multi-Source Integration: In Multi-Source Data Repairing: A Comprehensive Survey, Ye et al. show that when merging multiple databases, entity duplication, attribute conflicts, and logical inconsistencies co-occur and influence each other. Resolving one requires simultaneously resolving the others.
  • File Ingestion: The Pollock benchmark, introduced by Hameed et al., demonstrates that real-world CSV files routinely present multiple interacting syntactic pollutions that break standard parsers when they occur together.
  • String Semantics: In DataVinci: Learning Syntactic and Semantic String Repairs, Singh et al. point out that string data frequently suffers from a mix of syntactic errors (like formatting variations) and semantic errors (like incorrect named entities), which cannot be cleanly separated.
  • Textual Factuality: Even in natural language processing, Guo et al. demonstrate in Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency that a lone evaluation metric fails to clean text benchmarks because it misses the diverse range of co-existing factual errors, such as unverifiable content versus distorted semantic relationships.

Mechanism 1: Unifying Heterogeneous Rules

To clean mixed errors simultaneously, a system must first speak a single language. Real-world data quality constraints are specified in wildly different formats, ranging from formal mathematical expressions (like conditional functional dependencies) to informal, imperative code snippets embedded in business logic.

Formal Rule (Functional Dependency): 
   ZipCode -> State

Informal Rule (Business Logic Code): 
   if (age < 18 && driver_license == true) return error;

To bridge this gap, Ebaid et al. designed NADEEF: A Generalized Data Cleaning System. NADEEF works by compiling these heterogeneous rule types into a unified, abstract format. By translating both declarative constraints and procedural code into a common execution target, the underlying engine can detect violations holistically without caring about the original form of the specification.

Complementing this approach, Xu Chengyu proposed the Extensible Data Cleaning Framework (EDCF), which decouples the cleaning logic entirely into open rule and algorithm libraries. This architecture allows developers to plug in custom business rules alongside standard algorithms, ensuring the system remains flexible enough to handle a broad spectrum of formatting and logical errors without requiring a complete redesign for each new error type.

Mechanism 2: Joint Inference via Hybrid Modeling

Once rules are unified, the system must decide how to repair the data. Traditional systems separate qualitative, rule-based constraints from quantitative, statistical models. Modern frameworks combine them to achieve joint inference.

A prominent example of this strategy is MLNClean, introduced by Ge et al. in A Hybrid Data Cleaning Framework Using Markov Logic Networks. MLNClean leverages Markov Logic Networks (MLNs) to unify qualitative and quantitative cleaning into a two-stage probabilistic model:

$$\text{First-Order Logic Rules (Qualitative)} + \text{Data Distribution Weights (Quantitative)} \longrightarrow \text{Joint Probabilistic Graph}$$

By translating domain rules into first-order logic formulas and assigning weights based on statistical evidence in the dataset, MLNClean constructs a unified graph. The system then calculates reliability and fusion scores to resolve schema-level violations and instance-level anomalies (like typos or duplicates) at the same time. This prevents the sequential bias of older pipelines. The authors expanded on this integrated philosophy in IHCS: An Integrated Hybrid Cleaning System, tightly coupling error detection and repair within a single loop to handle diverse formats and error types concurrently.

Mechanism 3: Automated Workflow Orchestration

The final piece of the puzzle is scalability and automation. Even with unified models, manually configuring pipelines for massive datasets with shifting "dirtiness profiles" is unsustainable.

To address this, Ding et al. developed UniClean: A Scalable Data Cleaning Solution for Mixed Errors based on Unified Cleaners and Optimized Cleaning Workflow. UniClean introduces abstract "unified cleaners" that target multiple error categories simultaneously, using an optimized workflow engine to scale these operations across massive datasets without requiring extensive manual tuning.

[Raw Dataset] 
      │
      ▼
[LLM / Workflow Orchestrator] ──► Analyzes "Dirtiness Profile"
      │
      ▼
[Optimized Multi-Cleaner Pipeline] ──► Resolves Typos, Missing Values, & Rules Jointly
      │
      ▼
[Cleaned Dataset]

Taking automation a step further, Li et al. proposed AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark. Instead of relying on rigid, pre-written code, this approach leverages Large Language Models (LLMs) to dynamically analyze a dataset's unique combination of errors and automatically generate a tailored cleaning workflow. Because LLMs possess broad semantic understanding, they can flexibly adapt to unexpected inconsistencies, generating custom code and orchestration logic that addresses the complex, multi-type data corruption characteristic of real-world applications.

Key Foundational Work

NADEEF: A Generalized Data Cleaning System (2013) — contributed 3 concept elements to this direction.
Letter from the Special Issue Editor (2000) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lack_of_universal_data_cleaning_tools_c89a3ddb,
  title = {lack of universal data cleaning tools},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lack-of-universal-data-cleaning-tools/rdr_c89a3ddbeb9947e4a0b61b9c2c9eceae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_lack_of_universal_data_cleaning_tools_c89a3ddb_v1,
  title = {lack of universal data cleaning tools},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lack-of-universal-data-cleaning-tools/rdr_c89a3ddbeb9947e4a0b61b9c2c9eceae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake