Normalized Loss Functions for Deep Learning with Noisy Labels

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, James Bailey

TL;DR

This paper reveals a surprising theoretical insight: any loss function can be made robust to noisy labels simply by normalizing it. However, because robust losses often suffer from underfitting, the authors propose the Active Passive Loss (APL) framework. By combining two robust loss terms that balance each other, APL achieves state-of-the-art accuracy even when up to 80% of the training labels are incorrect.

The Problem: Overfitting vs. Underfitting

Deep Neural Networks (DNNs) are notoriously "label hungry," but high-quality, clean labels are expensive and difficult to obtain at scale. Real-world datasets often contain "noisy" (incorrect) labels.

Standard loss functions like Cross Entropy (CE) are highly sensitive to these errors. During training, the model essentially "memorizes" the noise, leading to a sharp degradation in generalization and test accuracy after an initial peak.

Test Accuracy vs Epochs for Standard and Normalized Losses

While researchers have developed robust alternatives like Mean Absolute Error (MAE), these often suffer from underfitting. They are so cautious about the noise that they fail to learn the actual patterns in the data effectively. The core challenge in learning with noisy labels is finding a loss function that is both robust to noise and "sufficient" for learning complex features.

Methods: Normalization and the APL Framework

The authors tackle this challenge in two distinct steps:

1. Normalization

They mathematically prove that if a loss function is normalized such that the sum of the loss over all possible classes is constant, it becomes noise-tolerant.

$$ \mathcal{L}_{norm} = \frac{\mathcal{L}(f(x), y)}{\sum_{j=1}^K \mathcal{L}(f(x), j)} $$

This simple transformation can turn non-robust losses (like CE or Focal Loss) into robust ones (Normalized Cross Entropy or Normalized Focal Loss).

2. The Active Passive Loss (APL) Framework

Recognizing that normalized losses often underfit (as seen in the flat, low-accuracy curves of robust losses), the authors categorize robust losses into two types:

  • Active Loss: Explicitly maximizes the probability of the model picking the labeled class (e.g., Normalized Cross Entropy).
  • Passive Loss: Explicitly minimizes the probability of the model picking the other (incorrect) classes (e.g., Reverse Cross Entropy or MAE).

The proposed Active Passive Loss (APL) framework combines one of each:

$$ \mathcal{L}_{APL} = \alpha \cdot \mathcal{L}_{Active} + \beta \cdot \mathcal{L}_{Passive} $$

This synergy allows the "Active" part to drive learning and convergence, while the "Passive" part maintains robustness and prevents the model from getting stuck in low-confidence states.

Key Insights

  • Robustness is "Cheap," Learning is Hard: The most surprising finding is that robustness isn't a rare property; it can be mathematically "forced" onto any loss via normalization. The real difficulty is ensuring the model still converges to a high-accuracy state.
  • The Underfitting Trap: Robust losses often have "flat" gradients. If the model is slightly unsure, the gradient becomes too small to drive further learning. This is why standard robust losses often fail on complex datasets.
  • Active-Passive Synergy: An Active loss alone might ignore the "background" classes, while a Passive loss alone might not push the correct class hard enough. By combining them, the APL framework ensures the model is always receiving a useful signal to distinguish the correct class from the noise.

Key Results

The authors evaluated their framework on standard benchmarks (MNIST, CIFAR-10, CIFAR-100) with synthetic noise, as well as on WebVision, a large-scale dataset with real-world noisy labels. APL consistently outperformed state-of-the-art (SOTA) methods like Generalized Cross Entropy (GCE) and Symmetric Cross Entropy (SCE):

  • Extreme Noise: On CIFAR-100 with 80% symmetric noise, APL (specifically NCE+RCE) achieved ~25.8% accuracy, while standard Cross Entropy collapsed to ~7.6%.
  • SOTA Comparison: On CIFAR-10 with 60% noise, APL reached ~79.8% accuracy, outperforming the previous SOTA (SCE) which sat at ~66%.
  • Real-World Performance: On the WebVision "Mini" dataset, APL achieved a top-1 validation accuracy of 62.64%, surpassing both GCE (53.68%) and SCE (61.76%).

Limitations and Future Work

  • Hyperparameter Sensitivity: While the authors suggest that $\alpha=1, \beta=1$ works well as a default, more complex datasets (like CIFAR-100) required tuning (e.g., $\alpha=10, \beta=0.1$) to perfectly balance the active and passive components.
  • Instance-Dependent Noise: The study primarily focuses on symmetric and asymmetric label noise. Future work could explore how APL handles "instance-dependent" noise, where the likelihood of a label being wrong depends on the specific features of the image itself.
  • Theoretical Bounds: While the paper provides proofs for symmetric and asymmetric noise, further theoretical work could explore the convergence rates of APL compared to standard losses.

Summary updated

Abstract

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss i...
BibTeX
@inproceedings{ma2020normalized,
  author = {Xingjun Ma and Hanxun Huang and Yisen Wang and Simone Romano and Sarah M. Erfani and James Bailey},
  title = {{Normalized Loss Functions for Deep Learning with Noisy Labels}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=PS_F_MOfkhP},
}
Figure