Abstract
Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibit lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the Jensen-Shannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-art results on both synthetic (CIFAR), and real-world (e.g., WebVision) noise with varying noise rates.
Introduction
Deep learning models are notoriously sensitive to label noise. While standard Cross-Entropy (CE) loss is efficient for clean data, it tends to "memorize" incorrect labels, leading to poor generalization. Conversely, noise-robust alternatives like Mean Absolute Error (MAE) are often difficult to optimize, resulting in underfitting. Researchers from KTH have introduced a new family of loss functions based on Generalized Jensen-Shannon (GJS) divergence that bridges this gap. GJS provides a single, principled framework that balances learnability and robustness while explicitly enforcing prediction consistency.
The GJS Framework: Bridging CE and MAE
The core innovation lies in using Jensen-Shannon (JS) divergence as a mathematical bridge. By adjusting a mixing parameter, the JS loss can interpolate between the fast-learning CE and the noise-robust MAE. This allows practitioners to tune the "aggressiveness" of the learning process based on the expected noise level.
The Generalized JS (GJS) loss extends this to handle multiple distributions simultaneously. By processing several augmented versions of the same image, the GJS loss naturally decomposes into a "fitting" term (matching the provided label) and a "consistency" term (ensuring similar predictions across augmentations). Unlike complex multi-stage pipelines, GJS is a drop-in replacement for standard loss functions.
Consistency as a Proxy for Label Quality
A key insight of this work is that prediction consistency serves as a reliable indicator of label correctness. Throughout training, clean data points maintain high consistency across different augmentations. In contrast, noisy data points exhibit a sharp drop in consistency as the model begins to overfit the incorrect label.
As shown in the figure above, while validation accuracy (a) remains stable or improves, the consistency for noisy samples (c) degrades significantly compared to clean samples (b) as epochs progress. This observation justifies the use of the GJS consistency term to penalize the model when it starts to memorize noise.
Performance Benchmarks
The GJS approach was evaluated against state-of-the-art methods on several benchmarks:
- CIFAR-100: With 60% random noise, GJS outperformed the next best robust loss (GCE) by nearly 5 percentage points.
- WebVision: On this real-world dataset (~20% noise), GJS achieved a top-1 accuracy of 79.28%, setting a new benchmark for models trained from scratch.
- Extreme Noise: GJS remained stable even at 80% symmetric noise, where standard CE accuracy plummeted.
Conclusion
The Generalized Jensen-Shannon divergence loss offers a theoretically grounded and practically simple solution to the problem of noisy labels. By leveraging the consistency signal and providing a tunable bridge between CE and MAE, it achieves superior robustness without the complexity of multi-stage training pipelines. Future work may explore the theoretical limits of the consistency term and optimize the computational overhead associated with multiple forward passes.
Summary updated