Active Negative Loss: A Robust Framework for Learning with Noisy Labels

Xichen Ye, Yifan Wu, Yiwen Xu, Xiaoqiang Li, Weizhong Zhang, Yifan Chen

Abstract

Deep supervised learning has achieved remarkable success across a wide range of tasks, yet it remains susceptible to overfitting when confronted with noisy labels. To address this issue, noise-robust loss functions offer an effective solution for enhancing learning in the presence of label noise. In this work, we systematically investigate the limitation of the recently proposed Active Passive Loss (APL), which employs Mean Absolute Error (MAE) as its passive loss function. Despite the robustness brought by MAE, one of its key drawbacks is that it pays equal attention to clean and noisy samples; this feature slows down convergence and potentially makes training difficult, particularly in large-scale datasets. To overcome these challenges, we introduce a novel loss function class, termed Normalized Negative Loss Functions (NNLFs), which serve as passive loss functions within the APL framework. NNLFs effectively address the limitations of MAE by concentrating more on memorized clean samples. By replacing MAE in APL with our proposed NNLFs, we enhance APL and present a new framework called Active Negative Loss (ANL). Moreover, in non-symmetric noise scenarios, we propose an entropy-based regularization technique to mitigate the vulnerability to the label imbalance. Extensive experiments demonstrate that the new loss functions adopted by our ANL framework can achieve better or comparable performance to state-of-the-art methods across various label noise types and in image segmentation tasks.

Introduction

Deep Neural Networks (DNNs) are notorious "memorizers." While they excel at learning patterns from clean data, they are also capable of fitting completely random labels if given enough time. In the real world, large-scale datasets often contain "noisy" labels due to human error or ambiguity. Standard loss functions like Cross Entropy (CE) are highly sensitive to this noise, causing models to memorize incorrect labels and perform poorly on actual test data.

Existing solutions like Mean Absolute Error (MAE) are theoretically robust but practically difficult to use: they converge slowly and treat every sample with equal importance. This means the model spends as much effort trying to learn a clearly mislabeled "noisy" sample as it does a "clean" one. The Active Negative Loss (ANL) framework addresses these inefficiencies by shifting the training focus from simply maximizing the probability of the labeled class to actively minimizing the probabilities of incorrect classes.

The ANL Framework

The authors propose the Active Negative Loss (ANL) framework, which replaces the inefficient MAE component in previous frameworks with a novel class of functions called Normalized Negative Loss Functions (NNLFs). The construction of an NNLF involves three key steps:

  1. Complementary Label Learning: Instead of asking "Is this a cat?", the model learns by asking "What is this not?" It focuses on the $K-1$ classes that are not the provided label.
  2. Vertical Flipping: This operation mathematically transforms an "active" loss (which maximizes the correct class probability) into a "passive" loss (which minimizes incorrect class probabilities).
  3. Normalization: The loss is divided by the sum of losses over all possible classes, a step theoretically shown to make almost any loss function robust to label noise.

Additionally, for "asymmetric noise" scenarios (where certain classes are more likely to be confused, like "dog" and "cat"), the authors introduce an entropy-based regularization term. This encourages the model's marginal output distribution to remain balanced, preventing it from becoming biased toward specific classes.

Figure 2: The effect of entropy-based regularization on class probability distributions under asymmetric noise. ANL-CE* (blue) maintains a more balanced distribution across categories compared to the asymmetric baseline (orange).

Experimental Results

The framework was evaluated on a comprehensive suite of benchmarks, including synthetic noise (CIFAR-10/100), real-world noise (WebVision, Animal-10N, Clothing-1M), and medical segmentation (ISIC-2017).

Training Dynamics and Robustness

A key advantage of ANL is its ability to focus gradients on samples the model has already started to "memorize" correctly. This allows the model to continue refining its knowledge of clean samples while effectively ignoring unmemorized noisy samples. As shown in the training curves, standard Cross Entropy (CE) quickly overfits to noisy samples, causing test accuracy to plummet. In contrast, ANL variants maintain high test accuracy by suppressing the memorization of noise.

Figure 3: Comparison of training dynamics across different loss functions. ANL-CE variants (d, e, i, j) demonstrate superior robustness, maintaining high test accuracy (blue line) while preventing the model from fitting noisy training samples (red dashed line).

Performance Highlights

  • CIFAR-10: With 80% symmetric noise, ANL-CE outperformed the previous state-of-the-art (APL) by over 5.7% in test accuracy.
  • Clothing-1M: ANL-CE achieved a top accuracy of 69.93%, surpassing other robust methods like GCE and SCE.
  • Medical Segmentation: In skin lesion segmentation (ISIC-2017), the framework maintained a high Dice score of 77.3% even under extreme noise conditions, proving its versatility beyond classification.

Conclusion

Active Negative Loss (ANL) provides a robust and efficient framework for training DNNs in the presence of significant label noise. By leveraging Normalized Negative Loss Functions and entropy-based regularization, ANL achieves state-of-the-art performance across diverse benchmarks. While highly robust, the authors note that proper regularization (such as L1) remains important for stabilizing training at extreme noise rates. Future work will focus on automating hyperparameter tuning to make the framework even more accessible for new datasets.

Summary updated

Abstract

Deep supervised learning has achieved remarkable success across a wide range of tasks, yet it remains susceptible to overfitting when confronted with noisy labels. To address this issue, noise-robust loss functions offer an effective solution for enhancing learning in the presence of label noise. In this work, we systematically investigate the limitation of the recently proposed Active Passive Loss (APL), which employs Mean Absolute Error (MAE) as its passive loss function. Despite the robustness brought by MAE, one of its key drawbacks is that it pays equal attention to clean and noisy samples; this feature slows down convergence and potentially makes training difficult, particularly in large-scale datasets. To overcome these challenges, we introduce a novel loss function class, termed Normalized Negative Loss Functions (NNLFs), which serve as passive loss functions within the APL framework. NNLFs effectively address the limitations of MAE by concentrating more on memorized clean samples. By replacing MAE in APL with our proposed NNLFs, we enhance APL and present a new framework called Active Negative Loss (ANL). Moreover, in non-symmetric noise scenarios, we propose an entropy-based regularization technique to mitigate the vulnerability to the label imbalance. Extensive experiments demonstrate that the new loss functions adopted by our ANL framework can achieve better or comparable performance to state-of-the-art methods across various label noise types and in image segmentation tasks. The source code is available at: https://github.com/Virusdoll/Active-Negative-Loss.
BibTeX
@misc{ye2024active,
  author = {Xichen Ye and Yifan Wu and Yiwen Xu and Xiaoqiang Li and Weizhong Zhang and Yifan Chen},
  title = {{Active Negative Loss: A Robust Framework for Learning with Noisy Labels}},
  year = {2024},
  eprint = {2412.02373},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.02373},
}
Figure