Abstract
Data quality is traditionally measured by the "3Cs": Consistency, Completeness, and Currency. These dimensions are deeply intertwined, yet most data-cleaning tools address them in isolation. Improve3C is a novel framework that leverages logical constraints to infer a relative temporal order (currency) among records, even when explicit timestamps are missing. By integrating this currency information into the repair process, Improve3C significantly enhances the accuracy of both consistency and completeness repairs.
Introduction
In modern data management, ensuring high data quality is a persistent challenge. Inconsistency (contradictory data), incompleteness (missing values), and outdatedness (stale data) often occur simultaneously. For instance, updating a missing "City" attribute using an old record might conflict with a more recent "Employer" entry. Traditional cleaning methods often fail because they lack a sense of temporal progression, especially in multi-source datasets where timestamps are frequently absent or unreliable.
Improve3C addresses this by treating currency not just as a problem to be solved, but as a foundational "glue" that informs the repair of other quality issues. By using domain-specific logical rules to establish a relative "freshness" order, the framework can distinguish between old and new data, leading to more reliable repairs.
Methodology
The Improve3C framework operates through a four-stage pipeline designed to progressively refine data quality.
1. Currency Graph Construction
The process begins by defining Currency Constraints (CCs)—logical rules provided by domain experts that describe how data evolves over time (e.g., a professional's "Job Level" typically increases). These constraints are used to build a directed currency graph where an edge from record $A$ to $B$ indicates that $B$ is more current than $A$.
2. Currency Order Determination
Using the CurrValue algorithm, the system assigns a numerical "currency value" between 0 and 1 to each record. This value is derived from the record's position within the longest chains of the currency graph, effectively creating a synthetic timestamp that represents relative freshness.
3. Inconsistency Repair (ImpCCons)
The framework identifies contradictions using Conditional Functional Dependencies (CFDs). When conflicts arise, Improve3C employs a "currency-related consistency distance" to select the most appropriate repair. It prioritizes values from records that are both logically consistent and temporally proximate to the record being repaired.
4. Incompleteness Repair (ImpCCom)
To fill missing values, Improve3C utilizes a Naive Bayes classifier. A key innovation here is the inclusion of the calculated currency value as a feature in the training model. This allows the classifier to learn temporal patterns, such as how salary or job titles change over a career trajectory, resulting in more accurate imputations.
Evaluation
The framework was evaluated using two datasets: a real-world NBA Player Statistics dataset (~25,000 records) and a synthetic Personal Career Information (PCI) dataset (40,000 records). Noise was introduced at rates ranging from 5% to 20% to test the system's robustness.
Improve3C demonstrated a significant performance boost over baseline methods (which combine standard CFD repair and Naive Bayes without currency awareness). Specifically, the framework improved precision by approximately 25% and recall by over 27%. On the NBA dataset, it achieved a precision exceeding 0.9. Furthermore, the system remained robust under high noise conditions, maintaining precision and recall above 0.84 even with 20% data corruption.
Conclusion
Improve3C demonstrates that logical currency constraints can effectively substitute for missing timestamps in data cleaning. By establishing a relative temporal order, the framework prevents the use of stale data for repairs, thereby improving both consistency and completeness. While the system's performance depends on the quality of provided constraints and faces some computational scaling challenges, it represents a significant step toward holistic data quality management. Future work will focus on parallelizing the graph construction to handle larger-scale "Big Data" environments.
Summary updated