Introduction
Most data-cleaning techniques handle "harmful" training samples—those with noisy labels or distribution biases—by simply throwing them away or downweighting them. However, discarding data loses the underlying information in the input features. This paper introduces RDIA (Resolving training biases via Influence-based DAta relabeling), a framework that uses influence functions to intelligently relabel these samples instead of discarding them. The authors theoretically and empirically prove that reusing fixed data leads to better model generalization and superior robustness against noise.
The Problem: Label Noise and Distribution Mismatch
Machine learning models are only as good as their training data. Real-world datasets are often plagued by two types of harmful samples:
- Label Noise: Samples where the assigned category is simply wrong (e.g., a "cat" image labeled as "dog").
- Distribution Mismatch: Samples that don't represent the actual test environment, leading the model to learn biases rather than useful features.
The standard industry response is data resampling: identifying these samples and either deleting them or reducing their importance during training. The challenge RDIA addresses is how to determine what the "correct" or "most helpful" label should be without manual inspection, allowing the model to retain the valuable input features.
The RDIA Framework
The core of RDIA relies on Influence Functions, a technique from robust statistics that estimates how much a specific training point affects the model's test performance without requiring the model to be retrained from scratch.
The RDIA workflow follows four main steps:
- Initial Training: Train a model on the original (potentially biased) dataset.
- Harmful Sample Identification: Use influence functions (evaluated against a clean validation set) to find samples that, if upweighted, would increase the test loss.
- Relabeling via Function $\mathcal{R}$: Instead of deleting these samples, the authors apply a novel relabeling function $\mathcal{R}$. For multi-class problems, this function reassigns labels based on the model’s current probability distribution to maximize the sample's positive influence on the test set.
- Retraining: Train the final model on the new dataset containing the relabeled samples.
For deep learning models where calculating influence functions is computationally expensive (due to the need to invert a massive Hessian matrix), the authors introduce RDIA-LS. This version uses training loss as a proxy to identify harmful samples (the "small-loss trick"), making the method scalable to large neural networks.
Key Insights and Results
The researchers evaluated RDIA across ten diverse public datasets spanning NLP, Computer Vision, and Click-Through Rate (CTR) prediction (including MNIST, CIFAR10/100, and Clothing1M).
- Relabeling > Discarding: The authors provide a mathematical proof showing that for any classification task using cross-entropy loss, there exists a relabeling strategy guaranteed to result in a lower test loss than simply deleting the harmful samples.
- Reusing "Hard" Samples: Harmful samples often contain features that the model has already begun to learn. By flipping the label rather than deleting the point, RDIA forces the model to "unlearn" the bias while still benefiting from the sample's feature information.
- Extreme Noise Robustness: One of the most striking insights is RDIA's stability. While standard models (ERM) see their performance collapse as label noise increases, RDIA’s test loss remains remarkably flat, even when up to 80% of the training labels are corrupted.
- Superior Accuracy: RDIA outperformed state-of-the-art resampling methods on 9 out of 10 datasets using logistic regression. On CIFAR10 with 80% noise, RDIA-LS achieved 25.35% accuracy, significantly higher than the baseline ERM (16.90%). On the Clothing1M dataset, which contains natural "wild" noise, RDIA-LS achieved an accuracy of 69.64%, outperforming all other loss-based resampling baselines.
Limitations and Future Work
- Validation Set Dependency: RDIA requires a small, clean, and unbiased validation set to calculate the influence functions accurately. If the validation set is also biased, the relabeling might be ineffective.
- Computational Complexity: Although RDIA-LS addresses this for deep learning, the "pure" RDIA method still struggles with the $O(NP)$ complexity of calculating the Hessian matrix for very large parameter sets.
- Transfer Learning: The current framework assumes the validation and test sets come from the same distribution. Extending this to transfer learning—where the target distribution is entirely different—remains a promising but unexplored direction.
Summary updated