Abstract
CerFix is a data cleaning system that moves beyond simple error detection to provide "certain fixes"—repairs that are mathematically guaranteed to be correct. By leveraging high-quality master data and a minimal amount of human validation, the system can automatically repair up to 80% of data entry errors while ensuring no new errors are introduced.
Introduction
Dirty data is a significant economic burden, costing businesses hundreds of billions of dollars annually. While many tools exist to detect errors using integrity constraints or ETL processes, fixing them remains a challenge. Traditional methods often rely on heuristics—essentially "best guesses"—to repair data. The danger is that a heuristic might fix one error while inadvertently introducing another, or change a correct value to an incorrect one due to a lack of context. In critical fields like medicine, these minor repair errors can be life-threatening. CerFix addresses the need for "certainty": if the system changes a value, that change must be guaranteed to reflect the true state of the world.
System Architecture and Methodology
CerFix introduces a framework based on three pillars:
- Master Data: A repository of high-quality, "golden" reference data used as the ground truth.
- Editing Rules (eRs): Unlike standard integrity constraints that only flag errors, editing rules specify how to fix a value using master data. For example, if a zip code is validated, the system can update the city and street to match the master record.
- Certain Regions: This core innovation identifies a minimal set of attributes that, once validated by a human, provides enough information for the system to guarantee a certain fix for the rest of the record.
The system architecture includes a Region Finder that pre-computes these regions and a Data Monitor—an interactive UI. The monitor guides the user to validate only the most "informative" fields (highlighted in yellow). Once confirmed, the Rule Engine automatically propagates the fixes to the rest of the tuple (highlighted in green).
The Interactive Repair Process
The system's efficiency stems from identifying "pivot points" in a data record. By validating a small subset of data, the system "unlocks" the ability to fix the rest automatically. This prevents "collateral damage" common in constraint-based cleaning, where a mismatch might lead to an incorrect update. CerFix ensures that if a specific attribute (like an area code) is validated, dependent attributes (like city) are updated to the correct values found in master data.
The system also includes an auditing module, providing transparency by showing which rules were triggered and which master data records were used for each fix.
Key Results
In experimental studies using UK customer information, the authors found that users only needed to manually validate approximately 20% of the data values. CerFix was able to automatically and certainly fix the remaining 80%.
Unlike previous heuristic-based methods, CerFix ensures the repair process is consistent and does not introduce new semantic errors. By ranking certain regions by size, the system proactively suggests the smallest possible set of attributes for the user to check, significantly reducing the human-in-the-loop burden.
Limitations and Future Work
The system's "certainty" is entirely dependent on the accuracy of the master data. If the reference repository contains errors, CerFix will propagate them. Additionally, editing rules are currently specified manually via a web interface; future work could focus on more automated rule discovery. Finally, the initial setup phase for schema mapping and rule definition can be labor-intensive for complex databases.
Summary updated