Mitigating Memorization of Noisy Labels by Clipping the Model Prediction

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, Yixuan Li

Abstract

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.

Introduction

Supervised learning typically operates under the assumption that training labels represent the absolute ground truth. However, real-world datasets are frequently "noisy," containing incorrect labels due to human error or automated data collection. Standard deep neural networks are notorious for "memorizing" these incorrect labels, which severely degrades their performance on clean test data.

The primary culprit is the standard Cross Entropy (CE) loss. Because CE is unbounded, it penalizes mistakes infinitely. If a model is confident in one class but the (noisy) label indicates another, the loss can approach infinity. To minimize this massive penalty, the model distorts its internal logic to accommodate the error, leading to poor generalization. While specialized "robust" loss functions exist, they often suffer from underfitting, where the model becomes too cautious to learn complex patterns. This paper introduces LogitClip, a method to make existing losses robust without sacrificing learning capacity.

Methodology: Logit Clipping

The core innovation of this work is LogitClip, a simple yet mathematically grounded modification to the model's output layer. Instead of redesigning the loss function itself, LogitClip constrains the logits ($z$)—the raw scores produced before the Softmax layer. Specifically, it clamps the $L_p$ norm of the logit vector to a constant threshold $\tau$:

$$ \text{clip}_\tau(z) = \begin{cases} \tau \cdot \frac{z}{\|z\|_p} & \text{if } \|z\|_p \geq \tau \\ z & \text{else} \end{cases} $$

Why LogitClip Works

  1. Bounded Loss: By capping the magnitude of the logits, the maximum possible loss is mathematically guaranteed to be finite. This prevents the model from experiencing "infinite" pressure to memorize a noisy label.
  2. Direction Preservation: LogitClip only scales the vector; it does not change its direction. This ensures the model's predicted class remains consistent while its "overconfidence" is reined in.
  3. Universality: LogitClip acts as a "universal adapter," enhancing the noise robustness of nearly any existing loss function, including Focal loss, MAE, and others.
Figure 1: The relationship between the clipping threshold $\tau$ and the resulting loss bound. As $\tau$ decreases, the loss becomes more tightly bounded, providing greater robustness against label noise.

Experimental Evaluation

The authors evaluated LogitClip across several benchmarks, including CIFAR-10/100 with simulated noise (Symmetric, Asymmetric, and Instance-dependent) and real-world noisy datasets like CIFAR-10N/100N and WebVision (2.4 million images).

Key Results

LogitClip demonstrates significant performance gains, particularly in challenging noise environments:

  • Massive Accuracy Boosts: On CIFAR-10 with "instance-dependent" noise, LogitClip improved standard Cross Entropy accuracy from 68.26% to 86.74%.
  • Synergy with Robust Losses: It enhances already robust losses; for instance, it boosted Normalized Cross Entropy (NCE) on CIFAR-10 from 73.53% to 87.59%.
  • State-of-the-Art Performance: On the large-scale WebVision dataset, CE + LogitClip outperformed complex state-of-the-art methods, achieving a top-1 validation accuracy of 65.12%.
Table 1: Test accuracy (%) on CIFAR-10 under various noise types. LogitClip (LC) consistently improves the performance of standard and robust loss functions across all noise scenarios.

Insights and Discussion

A critical finding is that clipping the norm of the logit vector is superior to clipping individual values (e.g., a ReLU6-style approach). Clipping individual values can alter the direction of the prediction, potentially damaging the model's decision boundaries. LogitClip avoids this by maintaining the relative relationship between class scores.

However, the method requires careful tuning of the threshold $\tau$. A threshold that is too low may lead to underfitting, as the model lacks the dynamic range to distinguish between similar classes in complex datasets. Theoretical proofs in the paper (Theorems 2.2 and 2.3) provide a framework for understanding how this threshold balances robustness and learning capacity.

Conclusion

LogitClip provides a simple, effective, and universal way to mitigate the memorization of noisy labels in deep learning. By bounding the loss at the logit level, it prevents over-tuning to errors while maintaining the high performance of standard loss functions. Future work may explore the application of logit clipping to related fields such as Out-of-Distribution (OOD) detection and model calibration.

Summary updated

Abstract

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.
BibTeX
@inproceedings{wei2023mitigating,
  author = {Hongxin Wei and Huiping Zhuang and Renchunzi Xie and Lei Feng and Gang Niu and Bo An and Yixuan Li},
  title = {{Mitigating Memorization of Noisy Labels by Clipping the Model Prediction}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=g0ofsq1NRL},
}
Figure