TL;DR
This paper introduces a novel ensemble method called EQPD that leverages "data quality matrices" to improve the accuracy of data mining models. By simulating multiple imperfect versions of the data based on known error patterns, EQPD can achieve better predictive performance than traditional single models or even bagging, especially when dealing with noisy or imperfect data.
What Problem Does This Paper Address?
In the real world, data is rarely perfect. Information-oriented organizations often treat their databases as products of a "manufacturing process," and like any manufacturing process, it can introduce errors. While some errors (like incorrect zip codes) are easily detectable and correctable, many are systemic and harder to fix. These systemic errors mean our database is just one flawed version of the truth. Traditional data mining approaches often ignore these inherent data imperfections, leading to sub-optimal predictive results. This paper aims to bridge this gap by explicitly incorporating knowledge about data quality flaws into the data mining process.
Methods
The core innovation here is the Ensemble of Quality-Matrix Perturbed Data (EQPD) approach. The authors view data as a "manufactured" product, where the true values are unknown, and our database is just one possible "version" or perturbation of these true values. Data quality matrices, often generated as part of quality assurance systems like Six-Sigma, quantify these known error patterns.
EQPD works by:
- Perturbing the Data: Using the data quality matrices, the algorithm generates multiple "perturbed" versions of the original training data. These perturbations mimic the known error types and their probabilities.
- Building Multiple Models: A separate data mining model (in this paper, decision trees) is trained on each of these perturbed datasets.
- Ensembling Predictions: The predictions from all these individual models are aggregated (e.g., by voting) to produce a final prediction.
This is akin to ensemble methods like bagging, but instead of random sampling of data, EQPD uses quality matrices to guide the data perturbation, aiming to create more representative versions of the underlying "true" data.
Datasets
The authors evaluated their approach using five common datasets from the UCI repository:
- Credit Screening
- Breast Cancer
- Contraceptive
- Flags
- Soyabean
These datasets were used in conjunction with simulated data quality matrices to test the EQPD method under various conditions.
Key Insights and Ideas
- Data Quality Matrices as a Source of Information: The paper highlights that data quality matrices, often used for process improvement, can be directly leveraged to enhance predictive modeling. They provide a quantifiable way to represent known data imperfections.
- Simulating Imperfection Improves Prediction: By explicitly modeling and simulating the known errors in the data, EQPD can create a more robust model. This suggests that acknowledging and working with data imperfections, rather than solely trying to eliminate them, can be beneficial.
- Variance Reduction is Key: The empirical results strongly suggest that EQPD's improvement in accuracy is primarily due to a reduction in the variance component of the model's error. This means the ensemble of models is less sensitive to the specific "flawed" version of the data it's trained on.
- Robustness to Imperfect Matrices: The approach shows resilience even when the quality matrices themselves are not perfectly accurate, still outperforming single models and often bagging.
Key Results
- Improved Accuracy: EQPD consistently outperformed a single decision tree model across all tested datasets.
- Outperforming Bagging: In many scenarios, particularly when data quality matrices had low variability (indicating more predictable error patterns), EQPD achieved better accuracy than bagging, even when using a similar number of models.
- Variance Reduction: Tables 6-8 demonstrate that EQPD significantly reduces the variance of the error compared to single models and bagging, providing a strong explanation for its improved performance. For instance, with correct matrices and low variability, EQPD reduced variance by up to 15% compared to bagging.
- Performance with Incorrect Matrices: Even when quality matrices were off by 25%, EQPD still performed better than a single model and often better than bagging, though the performance gains were less pronounced.
Limitations and Future Work
- Focus on Specific Dimensions: The current work primarily focuses on accuracy and contextual quality dimensions. The authors acknowledge that other dimensions like timeliness, completeness, and believability are also important but more challenging to model and incorporate.
- Number of Perturbations: The paper doesn't definitively determine the optimal number of perturbed versions (T) to build. Future work aims to establish bounds for this number, potentially by relating it to how close one wants to be to the "true" ensemble.
- Bias Reduction: While variance reduction is evident, the paper suggests further investigation is needed to determine if EQPD also reduces the bias component of error.
- Better Approximations: The current method of approximating data versions is a simplification. Future work will explore better approximation techniques.
Summary updated