Normalized Loss Functions for Deep Learning with Noisy Labels

Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, James Bailey

TL;DR

This paper introduces a simple normalization technique that makes any loss function robust to noisy labels. However, robustness alone isn't enough, as these normalized losses can underfit. To combat this, they propose the "Active Passive Loss" (APL) framework, which combines robust "active" and "passive" loss components, leading to state-of-the-art performance on noisy datasets.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) is notoriously sensitive to label noise – incorrect labels in the training data. The standard Cross-Entropy (CE) loss, while effective on clean data, performs poorly when labels are noisy. While many robust loss functions have been proposed, they often only offer partial robustness or suffer from other issues. This paper tackles two key challenges: 1) how to achieve guaranteed robustness for any loss function against label noise, and 2) how to overcome the underfitting problem observed in existing robust loss functions to ensure both robustness and accurate learning.

Methods

The core technical contributions are:

  1. Normalization for Robustness: The authors propose a simple normalization scheme for any loss function $L$ to create a normalized loss $L_{norm}$: $$L_{norm} = \frac{L(f(x), y)}{\sum_{j=1}^{K} L(f(x), j)}$$ They theoretically prove that this normalization makes any loss function robust to both symmetric and asymmetric label noise, provided the noise rate is not excessively high.

  2. Active Passive Loss (APL) Framework: Recognizing that normalized robust losses can underfit, the authors categorize loss functions into "Active" (explicitly maximizing the probability of the correct class) and "Passive" (explicitly minimizing probabilities of incorrect classes). They then propose the APL framework: $$L_{APL} = \alpha L_{Active} + \beta L_{Passive}$$ where $\alpha$ and $\beta$ are hyperparameters. This framework combines a robust active loss with a robust passive loss, aiming to achieve both robustness and sufficient learning capacity.

Datasets

The paper evaluates its methods on several standard datasets:

  • MNIST: A handwritten digit dataset.
  • CIFAR-10 / CIFAR-100: Image classification datasets with 10 and 100 classes, respectively.
  • WebVision: A large-scale, real-world noisy dataset for image classification.

The authors generate synthetic noisy labels by introducing symmetric (random flipping) and asymmetric (class-conditional flipping) noise at various rates (e.g., 20% to 80% for symmetric, 10% to 40% for asymmetric).

Key Insights and Ideas

  • Universally Robust Loss Functions: The most striking theoretical insight is that a simple normalization can make any loss function robust to label noise. This is a powerful generalization beyond specific robust loss designs.
  • Robustness Isn't Enough: The Underfitting Problem: Empirically, even theoretically robust loss functions (like normalized CE or MAE) can underperform because they struggle to learn complex patterns, leading to underfitting, especially on challenging datasets.
  • Active vs. Passive Learning: Categorizing losses into "Active" and "Passive" provides a new lens to understand their behavior. Active losses focus on maximizing correct predictions, while passive losses also actively suppress incorrect ones.
  • Synergy of Active and Passive: The APL framework demonstrates that combining these two types of losses is crucial. It addresses the underfitting of robust active losses by leveraging the explicit minimization of incorrect probabilities from passive losses, leading to better overall performance.

Key Results

  • Robustness Verified: Experiments confirm that normalizing non-robust losses like CE and FL makes them robust, preventing overfitting on noisy data.
  • APL Outperforms: The proposed APL framework consistently achieves state-of-the-art results across various datasets and noise levels. For instance, on CIFAR-100 with 80% symmetric noise, APL methods (NCE+MAE, NCE+RCE) achieved accuracies significantly higher (e.g., 9-10% improvement) than existing methods like GCE and NLNL.
  • WebVision Success: APL methods also showed strong performance on the challenging real-world noisy WebVision dataset, outperforming GCE and SCE.
  • Parameter Sensitivity: While APL generally performs well, optimal $\alpha$ and $\beta$ parameters can vary by dataset complexity, with more complex datasets (CIFAR-100) benefiting from a larger $\alpha$ (more active learning).

Limitations and Future Work

  • Underfitting of Scaled Losses: The paper shows that simply scaling normalized losses (e.g., NMAE, NRCE) does not fully resolve the underfitting issue, especially on complex datasets.
  • Hyperparameter Tuning: While APL generally performs well, finding the optimal $\alpha$ and $\beta$ requires some tuning, though the paper provides guidance based on dataset complexity.
  • Future Work: The authors suggest that their APL framework can serve as a general principle for developing new robust loss functions. They also hint that their theoretical insights could reshape loss function design towards properties beyond just robustness. The paper also mentions that GCE, while active, can be improved by incorporating a passive term within the APL framework.

Summary updated

Abstract

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.
BibTeX
@inproceedings{ma2020normalized,
  author = {Xingjun Ma and Hanxun Huang and Yisen Wang and Simone Romano and Sarah M. Erfani and James Bailey},
  title = {{Normalized Loss Functions for Deep Learning with Noisy Labels}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=PS_F_MOfkhP},
}
Figure