Research and Realization of the Extensible Data Cleaning Framework EDCF

Xu Chengyu, Wei GaoThe Open Automation and Control Systems Journal 2015 · DOI

Abstract

This paper proposes an extensible data cleaning framework (EDCF) based on key data cleaning technologies, featuring open rule and algorithm libraries. The framework allows for the cleaning of diverse data errors by applying specific business rules and appropriate algorithms. Experimental results verify the framework's efficiency and operational effectiveness in real-world scenarios.

Introduction

Data cleaning is a critical yet challenging task due to the variety of "dirty data," ranging from simple formatting errors to complex logical inconsistencies and near-duplicates. Traditional cleaning tools often suffer from a "one-size-fits-all" limitation, lacking the flexibility to incorporate domain-specific "prior knowledge" or adapt to evolving data requirements. The Extensible Data Cleaning Framework (EDCF) addresses these issues by decoupling cleaning logic into modular, extensible components.

The EDCF Architecture

The EDCF is designed around a four-stage workflow that ensures both automation and flexibility:

  1. Data Analysis: Users define cleaning rules and select algorithms tailored to the specific data source.
  2. Data Standardization: Utilizing a JDBC interface, the system ingests data and applies rules to unify formats (e.g., date patterns), which simplifies subsequent processing.
  3. Data Cleaning: A two-step process where outliers are addressed first, followed by the detection and merging of "approximately duplicated" records.
  4. Treatment & Verification: Results are presented for review, allowing for manual overrides and the generation of audit logs.

Core Components: Rules and Algorithms

The framework's extensibility is rooted in two primary libraries:

  • Open Rule Library: This library stores business logic, duplicate recognition thresholds, and merging rules. For instance, the RULE_TABLE defines how specific fields are validated and processed.
  • Algorithms Library: Built using object-oriented principles, this library allows new algorithms (e.g., fuzzy logic, clustering) to be integrated without modifying the core framework.
Structure of the RULE_TABLE in the EDCF Open Rule Library

Experimental Validation

To test the EDCF, the authors utilized a dataset from a foreign trade processing system containing 52,681 commodity records. To simulate challenging real-world conditions, the dataset was injected with 35% duplicate records and 20% random error data.

Key Findings

  • Efficiency through Standardization: Early unification of data formats significantly reduced the computational overhead for deduplication algorithms.
  • Precision via Prior Knowledge: The ability to define business rules allowed the system to identify errors that purely statistical methods would overlook.
  • Modular Growth: The class-based algorithm library proved that the system could easily evolve to include more advanced techniques.

Conclusion and Future Directions

The EDCF provides a robust, modular solution for complex data cleaning tasks. While the current implementation demonstrates high efficiency, future work will focus on increasing autonomy to reduce manual intervention and scaling the architecture for "Big Data" environments involving millions of records.

Summary updated

Abstract

This paper proposes the idea of establishing an extensible data cleaning framework which is based on the key technology of data cleaning, and the framework includes open rules library and algorithms library. This paper provides the descriptions of model principle and working process of the extensible data cleaning framework, and the validity of the framework is verified by experiment. When the data are being cleaned, all the errors in the data source can be cleaned according to the specific business by the predefined rules of the cleaning and choosing the appropriate algorithm. The last stage of the realization initially completes the basic functions of data cleaning module in the framework, and the framework which has good efficiency and operation effect is verified by the experiment.
BibTeX
@article{chengyu2015research,
  author = {Xu Chengyu and Wei Gao},
  title = {{Research and Realization of the Extensible Data Cleaning Framework EDCF}},
  journal = {The Open Automation and Control Systems Journal},
  year = {2015},
  doi = {10.2174/1874444301507012039},
}
Figure