Unifying Interdependent Data Cleaning Tasks

17 concepts9 papers1 proposal

Imagine you are cleaning a massive database of customer records. You notice two major problems: some entries contain typos or conflicting phone numbers, and many records are duplicates that need to be merged.

The traditional engineering approach treats this as an assembly line. First, you run a data repairing script to fix the typos. Next, you pass the cleaned data to an entity resolution algorithm to merge duplicate profiles. Finally, you run an imputation model to fill in missing values.

This sequential pipeline is intuitive, but it frequently fails. Fixing a typo in the first step alters the very attributes that the downstream deduplication algorithm relies on to match records. Conversely, merging two records might suddenly reveal that a previously "correct" phone number is actually a conflict that needs repairing. By treating these tasks in isolation, traditional pipelines get trapped in suboptimal local minima, introduce new inaccuracies, or fail to terminate entirely.

To solve this, a powerful paradigm has emerged: unified data cleaning. Instead of executing cleaning tasks sequentially, this approach treats error detection, data repairing, entity resolution, and imputation as a single, symbiotic cycle.

The Feedback Loop Between Repairing and Resolution

The core insight driving this research direction is that data quality issues are deeply coupled. A change made to resolve an individual data inconsistency fundamentally alters the parameters used to determine if two records represent the same real-world entity.

As Floris Geerts et al. highlighted in their foundational work on integrity constraints, attribute-level error correction and record-level object identification form a complex feedback loop. In A revival of integrity constraints for data cleaning, the authors demonstrated that context-specific rules, like Conditional Functional Dependencies, can bridge this gap, using the semantic context of the data to drive both error detection and record matching simultaneously.

When we force these tasks into isolated steps, we miss critical opportunities for mutual improvement. In REPLACE: A Logical Framework for Combining Collective Entity Resolution and Repairing, Meghyn Bienvenu et al. formalized this synergy. They showed that resolving an entity can automatically correct a data inconsistency because the merged record provides the missing context needed to resolve the error. Conversely, identifying an erroneous fact can unblock an entity merge that was previously blocked by conflicting data.

┌────────────────────────────────────────────────────────┐
│                                                        │
│             ┌───────────────┐                          │
│             │ Data Repair   │                          │
│             └───────┬───────┘                          │
│                     │ Updates attributes               │
│                     ▼                                  │
│             ┌───────────────┐                          │
│             │  Entity Res.  │                          │
│             └───────┬───────┘                          │
│                     │ Merges records & exposes conflicts│
│                     ▼                                  │
│                                                        │
└────────────────────────────────────────────────────────┘

To exploit this interplay, Wenfei Fan et al. introduced Entity Enhancing Rules (REEs) in Parallel Discrepancy Detection and Incremental Detection. By embedding machine learning classifiers directly within logic rules, the authors created a mechanism that detects semantic conflicts and entity mismatches at the same time, allowing the two tasks to actively influence and accelerate one another.

The Expanding Cycle of Interdependence

The need for unification extends far beyond just entity resolution and data repairing. Modern data pipelines must also handle schema mapping, missing data imputation, and evolving ontologies. Each added task introduces new dependencies:

To manage this multi-task complexity, recent architectures seek complete integration. For example, Zian Bao et al. developed the Rock framework in Rock: Cleaning Data with both ML and Logic Rules, which unifies entity resolution, conflict resolution, timeliness deduction, and missing data imputation into a single, cohesive process.

Why Joint Optimization Works

The fundamental reason unified data cleaning works is that it shifts the problem from local optimization to joint inference.

Historically, treating incomplete and inconsistent information as twin facets of a single reasoning problem dates back to early knowledge representation theory, such as the approximation lattices proposed by Christopher U. Habel in Plurals, Cardinalities, and Structures of Determination. When applied to modern databases, this unified view changes how the system evaluates a cleaning action.

In an isolated pipeline, a data repairing tool evaluates an update based purely on local constraints (e.g., "Does this value match the column type?"). In a unified framework, a hypothesis in one task acts as a soft constraint or a source of evidence for all other tasks.

For instance, if the system forms a high-confidence hypothesis that Record A and Record B belong to the same entity, that hypothesis instantly unlocks new information: the missing fields in Record A can be confidently imputed from Record B, and the conflicting addresses between them can be flagged for resolution. Instead of generating conflicting updates that degrade data quality, the unified system coordinates its actions, converging on a globally consistent state that maximizes the accuracy of the entire dataset.

Key Foundational Work

A revival of integrity constraints for data cleaning (2008) — contributed 2 concept elements to this direction.
Plurals, Cardinalities, and Structures of Determination (1986) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interdependent_data_cleaning_tasks_15824b86,
  title = {interdependent data cleaning tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interdependent-data-cleaning-tasks/rdr_15824b8667ec46f1983373af528803da},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_interdependent_data_cleaning_tasks_15824b86_v1,
  title = {interdependent data cleaning tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interdependent-data-cleaning-tasks/rdr_15824b8667ec46f1983373af528803da/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake