Abstract
Deep learning models often memorize noisy labels, leading to degraded generalization. This paper introduces ANTIDOTE (Adversarial Neural-Network Training with Information Divergence-Reweighted Objective), a principled "plug-and-play" loss function that allows models to "forget" mislabeled data. By formulating training as a relaxation over information-divergence neighborhoods and utilizing convex duality, ANTIDOTE transforms complex optimization into an efficient objective. Experimental results across synthetic and real-world noisy datasets demonstrate that ANTIDOTE outperforms state-of-the-art robust losses with minimal added training time.
Introduction
Deep neural networks are highly effective at fitting training data, but this capacity often extends to memorizing errors. In real-world scenarios, datasets frequently contain "noisy labels" from human error or automated scraping. When a model encounters such noise, it eventually forces its parameters to fit these errors—a phenomenon known as noise memorization. Existing solutions typically involve expensive label correction or multi-stage pipelines. ANTIDOTE addresses this by providing a robust loss function that handles noise automatically without requiring clean data or complex training schemes.
Methodology
The core of ANTIDOTE is a relaxation-optimization strategy. Instead of minimizing loss on a fixed training distribution, the model searches for a "nearby" distribution that minimizes expected loss. "Nearness" is defined using f-divergences (e.g., KL-divergence or $\alpha$-divergence).
Key technical components include:
- Min-Min Optimization: Unlike adversarial training (min-max), which focuses on worst-case perturbations, ANTIDOTE uses a min-min approach to identify and downweight samples that deviate from majority patterns.
- Convex Duality: The authors use convex duality to simplify high-dimensional distribution searches into efficient 1D or 2D optimization problems.
- Dynamic Weighting: The loss function adaptively reweights samples. In the $\alpha$-divergence variant, the model establishes a "forgetting threshold," completely ignoring samples with loss exceeding this value.
Experimental Setup
ANTIDOTE was evaluated on several benchmarks:
- Synthetic Noise: CIFAR-10 and CIFAR-100 with up to 80% symmetric and asymmetric label noise.
- Human-Annotated Noise: CIFAR-100N.
- Large-Scale Real-World Noise: WebVision and ILSVRC12 (ImageNet) validation sets.
Insights and Analysis
A key insight of the paper is that "forgetting" is a necessary feature for learning from noisy data. By providing a "forgetting budget" ($\delta$), the model can isolate likely errors.
The authors demonstrate that as training progresses, the loss distributions of clean and noisy samples naturally diverge. This separation allows the "forgetting threshold" to effectively filter out noise.
As shown in the figures above, the overlap between clean and noisy label distributions decreases over time, facilitating the identification of erroneous samples.
Results
- Accuracy Gains: ANTIDOTE consistently outperformed baselines. On CIFAR-10 with 60% symmetric noise, it achieved 83.17% accuracy (vs. ~80.36% for the next best).
- Robustness to High Noise: The performance gap widens in high-noise regimes. In 40% asymmetric noise on CIFAR-10, ANTIDOTE improved accuracy by over 5%.
- Scalability: On ImageNet, ANTIDOTE improved top-1 accuracy by 1.12% over comparable baselines.
- Efficiency: Training time overhead is negligible. For CIFAR-100, ANTIDOTE added only 24 seconds to a ~33-minute training run.
Discussion and Future Work
While effective, ANTIDOTE requires tuning the neighborhood size parameter ($\delta$). Although the authors suggest heuristics for estimation, future work could focus on making this parameter fully adaptive. Additionally, while this study focused on Cross-Entropy loss, the framework could be extended to other objectives like ranking or self-supervised learning.