Improve3C: Data Cleaning on Consistency and Completeness with Currency

Xiaoou Ding, Hongzhi Wang, Jiaxuan Su, Jianzhong LiarXiv 2018 · arXiv

Abstract

Data quality is traditionally measured by the "3Cs": Consistency, Completeness, and Currency. These dimensions are deeply intertwined, yet most data-cleaning tools address them in isolation. Improve3C is a novel framework that leverages logical constraints to infer a relative temporal order (currency) among records, even when explicit timestamps are missing. By integrating this currency information into the repair process, Improve3C significantly enhances the accuracy of both consistency and completeness repairs.

Introduction

In modern data management, ensuring high data quality is a persistent challenge. Inconsistency (contradictory data), incompleteness (missing values), and outdatedness (stale data) often occur simultaneously. For instance, updating a missing "City" attribute using an old record might conflict with a more recent "Employer" entry. Traditional cleaning methods often fail because they lack a sense of temporal progression, especially in multi-source datasets where timestamps are frequently absent or unreliable.

Improve3C addresses this by treating currency not just as a problem to be solved, but as a foundational "glue" that informs the repair of other quality issues. By using domain-specific logical rules to establish a relative "freshness" order, the framework can distinguish between old and new data, leading to more reliable repairs.

Methodology

The Improve3C framework operates through a four-stage pipeline designed to progressively refine data quality.

The Improve3C system architecture, illustrating the four-stage cleaning pipeline from preprocessing to final high-quality output.

1. Currency Graph Construction

The process begins by defining Currency Constraints (CCs)—logical rules provided by domain experts that describe how data evolves over time (e.g., a professional's "Job Level" typically increases). These constraints are used to build a directed currency graph where an edge from record $A$ to $B$ indicates that $B$ is more current than $A$.

2. Currency Order Determination

Using the CurrValue algorithm, the system assigns a numerical "currency value" between 0 and 1 to each record. This value is derived from the record's position within the longest chains of the currency graph, effectively creating a synthetic timestamp that represents relative freshness.

Examples of initial currency constraints (E1, E2) being transformed into determined currency values (Gc1, Gc2) through graph analysis.

3. Inconsistency Repair (ImpCCons)

The framework identifies contradictions using Conditional Functional Dependencies (CFDs). When conflicts arise, Improve3C employs a "currency-related consistency distance" to select the most appropriate repair. It prioritizes values from records that are both logically consistent and temporally proximate to the record being repaired.

4. Incompleteness Repair (ImpCCom)

To fill missing values, Improve3C utilizes a Naive Bayes classifier. A key innovation here is the inclusion of the calculated currency value as a feature in the training model. This allows the classifier to learn temporal patterns, such as how salary or job titles change over a career trajectory, resulting in more accurate imputations.

Evaluation

The framework was evaluated using two datasets: a real-world NBA Player Statistics dataset (~25,000 records) and a synthetic Personal Career Information (PCI) dataset (40,000 records). Noise was introduced at rates ranging from 5% to 20% to test the system's robustness.

Improve3C demonstrated a significant performance boost over baseline methods (which combine standard CFD repair and Naive Bayes without currency awareness). Specifically, the framework improved precision by approximately 25% and recall by over 27%. On the NBA dataset, it achieved a precision exceeding 0.9. Furthermore, the system remained robust under high noise conditions, maintaining precision and recall above 0.84 even with 20% data corruption.

Conclusion

Improve3C demonstrates that logical currency constraints can effectively substitute for missing timestamps in data cleaning. By establishing a relative temporal order, the framework prevents the use of stale data for repairs, thereby improving both consistency and completeness. While the system's performance depends on the quality of provided constraints and faces some computational scaling challenges, it represents a significant step toward holistic data quality management. Future work will focus on parallelizing the graph construction to handle larger-scale "Big Data" environments.

Summary updated

Abstract

Data quality plays a key role in big data management today. With the explosive growth of data from a variety of sources, the quality of data is faced with multiple problems. Motivated by this, we study the multiple data quality improvement on completeness, consistency and currency in this paper. For the proposed problem, we introduce a 4-step framework, named Improve3C, for detection and quality improvement on incomplete and inconsistent data without timestamps. We compute and achieve a relative currency order among records derived from given currency constraints, according to which inconsistent and incomplete data can be repaired effectively considering the temporal impact. For both effectiveness and efficiency consideration, we carry out inconsistent repair ahead of incomplete repair. Currency-related consistency distance is defined to measure the similarity between dirty records and clean ones more accurately. In addition, currency orders are treated as an important feature in the training process of incompleteness repair. The solution algorithms are introduced in detail with examples. A thorough experiment on one real-life data and a synthetic one verifies that the proposed method can improve the performance of dirty data cleaning with multiple quality problems which are hard to be cleaned by the existing approaches effectively.
BibTeX
@misc{ding2018improve,
  author = {Xiaoou Ding and Hongzhi Wang and Jiaxuan Su and Jianzhong Li and Hong Gao},
  title = {{Improve3C: Data Cleaning on Consistency and Completeness with Currency}},
  year = {2018},
  eprint = {1808.00024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.00024},
}
Figure