Rapid Identification of Column Heterogeneity

Bing Tian Dai, Nick Koudas, Beng Chin Ooi, Divesh Srivastava, Suresh Venkatasubramanian

Abstract

Data quality is a serious concern in every data management application, and a variety of quality measures have been proposed, e.g., accuracy, freshness and completeness, to capture common sources of data quality degradation. We identify and focus attention on a novel measure, column heterogeneity, that seeks to quantify the data quality problems that can arise when merging data from different sources. We identify desiderata that a column heterogeneity measure should intuitively satisfy, and describe our technique to quantify database column heterogeneity based on using a novel combination of cluster entropy and soft clustering. Finally, we present detailed experimental results, using diverse data sets of different types, to demonstrate that our approach provides a robust mechanism for identifying and quantifying database column heterogeneity.
BibTeX
@inproceedings{dai2006rapid,
  author = {Bing Tian Dai and Nick Koudas and Beng Chin Ooi and Divesh Srivastava and Suresh Venkatasubramanian},
  title = {{Rapid Identification of Column Heterogeneity}},
  booktitle = {ICDM},
  year = {2006},
  doi = {10.1109/icdm.2006.132},
}

Loading related content...

Figure