Imagine you are cleaning a massive database of customer records. You notice two major problems: some entries contain typos or conflicting phone numbers, and many records are duplicates that need to be merged.
The traditional engineering approach treats this as an assembly line. First, you run a data repairing script to fix the typos. Next, you pass the cleaned data to an entity resolution algorithm to merge duplicate profiles. Finally, you run an imputation model to fill in missing values.
This sequential pipeline is intuitive, but it frequently fails. Fixing a typo in the first step alters the very attributes that the downstream deduplication algorithm relies on to match records. Conversely, merging two records might suddenly reveal that a previously "correct" phone number is actually a conflict that needs repairing. By treating these tasks in isolation, traditional pipelines get trapped in suboptimal local minima, introduce new inaccuracies, or fail to terminate entirely.
To solve this, a powerful paradigm has emerged: unified data cleaning. Instead of executing cleaning tasks sequentially, this approach treats error detection, data repairing, entity resolution, and imputation as a single, symbiotic cycle.
The Feedback Loop Between Repairing and Resolution
The core insight driving this research direction is that data quality issues are deeply coupled. A change made to resolve an individual data inconsistency fundamentally alters the parameters used to determine if two records represent the same real-world entity.
As Floris Geerts et al. highlighted in their foundational work on integrity constraints, attribute-level error correction and record-level object identification form a complex feedback loop. In A revival of integrity constraints for data cleaning, the authors demonstrated that context-specific rules, like Conditional Functional Dependencies, can bridge this gap, using the semantic context of the data to drive both error detection and record matching simultaneously.
When we force these tasks into isolated steps, we miss critical opportunities for mutual improvement. In REPLACE: A Logical Framework for Combining Collective Entity Resolution and Repairing, Meghyn Bienvenu et al. formalized this synergy. They showed that resolving an entity can automatically correct a data inconsistency because the merged record provides the missing context needed to resolve the error. Conversely, identifying an erroneous fact can unblock an entity merge that was previously blocked by conflicting data.
┌────────────────────────────────────────────────────────┐
│ │
│ ┌───────────────┐ │
│ │ Data Repair │ │
│ └───────┬───────┘ │
│ │ Updates attributes │
│ ▼ │
│ ┌───────────────┐ │
│ │ Entity Res. │ │
│ └───────┬───────┘ │
│ │ Merges records & exposes conflicts│
│ ▼ │
│ │
└────────────────────────────────────────────────────────┘
To exploit this interplay, Wenfei Fan et al. introduced Entity Enhancing Rules (REEs) in Parallel Discrepancy Detection and Incremental Detection. By embedding machine learning classifiers directly within logic rules, the authors created a mechanism that detects semantic conflicts and entity mismatches at the same time, allowing the two tasks to actively influence and accelerate one another.
The Expanding Cycle of Interdependence
The need for unification extends far beyond just entity resolution and data repairing. Modern data pipelines must also handle schema mapping, missing data imputation, and evolving ontologies. Each added task introduces new dependencies:
- Schema Mapping and Data Exchange: In That's All Folks! LLUNATIC Goes Open Source, Giansalvatore Mecca et al. demonstrated that pipelining data exchange followed by data cleaning fails because schema mappings introduce new inconsistencies, while data quality rules alter how those very mappings should be applied.
- Ontology Alignment: Data errors are often indistinguishable from gaps in our knowledge bases. Zheng Zheng et al. showed in Contextual Data Cleaning with Ontology Functional Dependencies that data inconsistencies frequently stem from incomplete ontologies rather than just bad entries. Their framework repairs the data and the underlying ontology simultaneously to prevent false positives.
- Imputation and Deep Learning: In a comprehensive survey, Jingyu Zhu et al. argued in Relational Data Cleaning Meets Artificial Intelligence: A Survey that treating error detection, repairing, and imputation as independent tasks leads to conflicting updates. This is particularly dangerous for downstream applications; as Yuji Roh et al. noted in Data collection and quality challenges in deep learning: a data-centric AI perspective, these isolated errors propagate and compound throughout deep learning pipelines.
To manage this multi-task complexity, recent architectures seek complete integration. For example, Zian Bao et al. developed the Rock framework in Rock: Cleaning Data with both ML and Logic Rules, which unifies entity resolution, conflict resolution, timeliness deduction, and missing data imputation into a single, cohesive process.
Why Joint Optimization Works
The fundamental reason unified data cleaning works is that it shifts the problem from local optimization to joint inference.
Historically, treating incomplete and inconsistent information as twin facets of a single reasoning problem dates back to early knowledge representation theory, such as the approximation lattices proposed by Christopher U. Habel in Plurals, Cardinalities, and Structures of Determination. When applied to modern databases, this unified view changes how the system evaluates a cleaning action.
In an isolated pipeline, a data repairing tool evaluates an update based purely on local constraints (e.g., "Does this value match the column type?"). In a unified framework, a hypothesis in one task acts as a soft constraint or a source of evidence for all other tasks.
For instance, if the system forms a high-confidence hypothesis that Record A and Record B belong to the same entity, that hypothesis instantly unlocks new information: the missing fields in Record A can be confidently imputed from Record B, and the conflicting addresses between them can be flagged for resolution. Instead of generating conflicting updates that degrade data quality, the unified system coordinates its actions, converging on a globally consistent state that maximizes the accuracy of the entire dataset.
Key Foundational Work
A revival of integrity constraints for data cleaning (2008) — contributed 2 concept elements to this direction.
Plurals, Cardinalities, and Structures of Determination (1986) — contributed 1 concept elements to this direction.