Introduction
The "Garbage In, Garbage Out" principle remains a fundamental challenge in machine learning: a model's performance is strictly bounded by the quality of its training data. Real-world datasets are frequently compromised by random noise, human labeling errors, or sophisticated adversarial "poisoning" and "backdoor" attacks.
Existing Data Quality Management (DQM) approaches typically fall into two categories. ML-oblivious methods focus on database-level consistency without regard for downstream model performance. Conversely, data valuation methods like Shapley values attempt to quantify the "price" of individual data points but are often computationally prohibitive and lack robust theoretical guarantees. To bridge this gap, researchers have introduced DATASIFTER, a task-driven, model-agnostic framework designed to identify and remove "bad" data by learning the characteristics of "good" data for specific machine learning tasks.
Methodology: The DATASIFTER Framework
DATASIFTER treats DQM as an optimization problem, operating in two distinct phases: the Learning Phase and the Selection Phase.
- The Learning Phase: Rather than calculating the value of every individual point, DATASIFTER learns a "utility model." It begins by randomly sampling subsets of the training data and training a "proxy" model on each. These models are evaluated against a small, clean validation set to generate (subset, performance score) pairs. These pairs are then used to train a DeepSets neural network, which learns to predict the utility of any given collection of data.
- The Selection Phase: Once trained, the utility model is used alongside a linear-time stochastic greedy algorithm to sift through the full dataset. The algorithm selects the subset predicted to yield the highest performance for the specific downstream task.
Theoretical Insights: The Shapley Trap
A significant contribution of this work is the Domination Analysis, a theoretical framework for comparing DQM strategies. The authors demonstrate that popular "linear heuristics" like Shapley values can fail in practical scenarios. Specifically, because Shapley values average a point's contribution across all subset sizes, they tend to over-weight performance on very small datasets. In datasets with high redundancy, this can lead to the selection of redundant points, resulting in poor diversity and suboptimal model performance.
Empirical Results
DATASIFTER was evaluated across diverse domains, including image classification (CIFAR-10, MNIST), text analysis (Enron1 SPAM), and medical imaging (COVID-CT).
The framework consistently outperformed state-of-the-art techniques across several DQM tasks:
- Adversarial Detection: In Trojan attack scenarios on CIFAR-10, DATASIFTER achieved a 90% detection rate while inspecting 51.17% fewer data points than the leading baseline.
- Noise Identification: It identified 90% of corrupted images by examining 67.25% fewer points than competing methods.
- Data Summarization and Debiasing: On the PubFig83 dataset, DATASIFTER selected representative subsets that maintained higher accuracy than random or Shapley-based selection. It also effectively filtered data to improve model fairness on the Adult Census and COMPAS datasets.
Conclusion and Future Work
While DATASIFTER represents a significant advance in DQM, it relies on the availability of a small, "gold standard" clean validation set. Additionally, while more efficient than exact Shapley calculations, the initial learning phase still involves training numerous proxy models. Future research may focus on more sophisticated set-function learning models beyond the DeepSets architecture and further reducing the computational overhead for massive datasets.
Summary updated