Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels

Erik Englesson, Hossein Azizpour

TL;DR

This paper introduces a novel loss function based on Generalized Jensen-Shannon (GJS) divergence to train deep neural networks on datasets with noisy labels. By cleverly interpolating between standard cross-entropy and robust mean absolute error, and incorporating a "consistency" term for perturbed inputs, GJS achieves state-of-the-art performance on both synthetic and real-world noisy datasets.

What Problem Does This Paper Address?

Deep learning thrives on massive labeled datasets, but real-world labels are often "noisy"—meaning some are incorrect due to human error or automated tagging mistakes. Standard training using Cross-Entropy (CE) loss is problematic because the network eventually "memorizes" these incorrect labels, leading to poor generalization.

While researchers have developed "noise-robust" loss functions like Mean Absolute Error (MAE), these often suffer from an "underfitting" problem: they converge very slowly and struggle to learn complex patterns. The authors identify a second crucial issue: networks trained with CE show significantly lower consistency (predicting different classes for slightly different versions of the same image) when dealing with noisy data points compared to clean ones.

Methods

The authors propose two related loss functions:

  1. Jensen-Shannon (JS) Loss: They prove that JS divergence acts as a bridge between CE and MAE. By adjusting a mixing parameter ($\pi_1$), the loss can behave like the fast-converging CE early on or the noise-robust MAE.
  2. Generalized Jensen-Shannon (GJS) Loss: This extends the JS loss to handle multiple distributions. In practice, this means the model looks at several differently augmented (perturbed) versions of the same image.

The GJS loss can be mathematically decomposed into two parts: a term that encourages the average prediction to match the (potentially noisy) label, and a consistency regularization term that forces the model to give the same answer for all versions of the image, regardless of the label.

Datasets

The researchers evaluated their method on several benchmarks:

  • CIFAR-10 & CIFAR-100: Used for synthetic noise tests, where labels were intentionally flipped at various rates (up to 80% noise).
  • WebVision: A large-scale real-world dataset containing images crawled from the web with an estimated 20% natural label noise.
  • ILSVRC12 (ImageNet): Used to validate the models trained on WebVision.

Key Insights and Ideas

  • Loss Interpolation: The JS loss provides a controllable "slider" between CE and MAE. This allows practitioners to find a sweet spot that converges quickly without overfitting to noise.
JS loss generalizes CE and MAE: Adjusting the hyperparameter π1 allows the loss to transition from CE-like behavior to MAE-like behavior.
  • The Consistency-Noise Link: The authors observed that as a network starts to overfit to noisy labels, its consistency on those noisy points drops sharply. This insight motivates using consistency as a signal to resist noise.
  • Principled Integration: Unlike previous methods that "tack on" consistency as an extra penalty term with its own weight, GJS incorporates consistency naturally within a single information-theoretic framework.

Key Results

The GJS loss consistently outperformed existing robust loss functions (like GCE and SCE) across different noise levels:

  • High Noise Performance: On CIFAR-100 with 60% symmetric noise, GJS achieved 70.15% accuracy, significantly outperforming the next best method (GCE at 65.21%).
  • Real-World Success: On the WebVision dataset, GJS achieved a 79.28% top-1 accuracy, which is among the highest reported results for a model trained without pre-training or complex multi-stage pipelines.
  • Stability: While CE loss accuracy and consistency crash as the model overfits, GJS remains stable and maintains high consistency throughout training.
Evolution of validation accuracy and consistency over epochs. GJS maintains high consistency and accuracy even as training progresses on noisy data.

Limitations and Future Work

  • Theoretical Gap: While the authors prove the robustness of the JS term, a full theoretical proof for the robustness of the consistency term in GJS remains an open challenge.
  • Diminishing Returns with More Augmentations: Interestingly, using more than three distributions ($M > 3$) actually degraded performance at very high noise rates. The authors suggest that at high noise levels, the consistency term might become too dominant, making it "easier" for the model to simply predict a uniform distribution rather than learning from the few clean labels available.
  • Computational Cost: GJS requires multiple forward passes (for different augmentations) during training, which increases training time, though it has no impact on inference speed.

Summary updated

Abstract

Prior works have found it beneficial to combine provably noise-robust loss functions e.g., mean absolute error (MAE) with standard categorical loss function e.g. cross entropy (CE) to improve their learnability. Here, we propose to use Jensen-Shannon divergence as a noise-robust loss function and show that it interestingly interpolate between CE and MAE with a controllable mixing parameter. Furthermore, we make a crucial observation that CE exhibit lower consistency around noisy data points. Based on this observation, we adopt a generalized version of the Jensen-Shannon divergence for multiple distributions to encourage consistency around data points. Using this loss function, we show state-of-the-art results on both synthetic (CIFAR), and real-world (e.g., WebVision) noise with varying noise rates.
BibTeX
@inproceedings{englesson2021generalized,
  author = {Erik Englesson and Hossein Azizpour},
  title = {{Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=TiwPYwg3IRf},
}
Figure