Mitigating Memorization of Noisy Labels by Clipping the Model Prediction

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, Yixuan Li

TL;DR

Standard Cross Entropy (CE) loss forces deep neural networks to memorize incorrect labels because its mathematical formulation is unbounded. By simply clipping the norm of the model's output logits (LogitClip), this paper introduces a universal, plug-and-play constraint that prevents networks from overfitting to label noise while consistently boosting generalization performance across various architectures and datasets.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are notoriously good at memorizing things—including mistakes. When trained on datasets containing "noisy" (incorrect) labels, standard Cross Entropy (CE) loss forces the model to confidently predict these wrong labels. Why? Because CE loss is unbounded; if the model disagrees with a training label, the loss approaches infinity. To avoid this massive penalty, the model warps its decision boundaries to overfit the noise, severely damaging its ability to generalize to clean test data.

While researchers have designed specialized "robust" loss functions to combat this, these alternatives often introduce optimization difficulties, leading to underfitting on complex datasets. This paper tackles a fundamental question: Can we make existing loss functions noise-robust without changing the loss functions themselves?

Methods

Instead of altering the loss function, the authors propose Logit Clipping (LogitClip), which bounds the loss at the logit level (the raw, unnormalized outputs of the model before the softmax layer).

Mathematically, LogitClip clamps the $\ell_p$-norm of the logit vector $z$ to a maximum threshold $\tau$:

$$ \text{clip}_\tau(z) = \begin{cases} \tau \cdot \frac{z}{\|z\|_p} & \text{if } \|z\|_p \ge \tau \\ z & \text{else} \end{cases} $$

Why Clip by Norm?

Unlike standard value clipping (e.g., clamping individual logit values between $[-6, 6]$), clipping the norm preserves the relative proportions and direction of the logit vector. This means the model's final class prediction remains completely unchanged, but its maximum possible loss is strictly capped.

Figure 1: The effect of τ and K on the Loss bound. The dashed lines denote the upper bounds and the solid lines show the lower bounds. Four colors are used to present bounds with various values of K.

The authors provide theoretical proofs showing that equipping CE loss with LogitClip guarantees strict upper and lower bounds on the loss, effectively preventing the gradient from exploding when encountering a noisy label.

Datasets

The authors evaluated LogitClip across a comprehensive suite of benchmarks with both simulated and real-world label noise:

  • CIFAR-10 & CIFAR-100: Evaluated using three simulated noise types: Symmetric (random flipping), Asymmetric (flipping to similar classes), and Instance-dependent (noise based on image features).
  • CIFAR-10N & CIFAR-100N: Human-annotated real-world noisy versions of the CIFAR datasets.
  • WebVision: A large-scale, real-world noisy dataset containing 2.4 million web images.

Key Insights and Ideas

  • Logit-Level vs. Loss-Level Modification: Instead of engineering complex new loss functions, controlling the magnitude of the network's raw outputs is a simpler, more direct way to enforce loss boundedness.
  • The Direction Matters: Value-clipping methods (like ReLU6) alter the direction of the logit vector, which can inadvertently change the model's predictions and zero out gradients. Norm-clipping maintains the model's predictive intent while safely managing its confidence.
  • The Goldilocks Zone of $\tau$: The hyperparameter $\tau$ controls the noise-tolerance threshold. As shown in Figure 2, a smaller $\tau$ (larger $1/\tau$) yields tighter theoretical risk bounds against noise. However, if $\tau$ is too small, the loss bounds collapse together, making optimization difficult and causing the model to underfit even clean data.
Figure 2: The Effect of τ in LogitClip with CIFAR-10 and CIFAR-100 across various noise types.

Key Results

LogitClip achieved massive performance gains across the board, proving to be both model-agnostic and loss-agnostic.

  • Revitalizing Cross Entropy: On CIFAR-10 with instance-dependent noise, adding LogitClip skyrocketed the test accuracy of standard CE loss from 68.36% to 86.74% (an 18.38% absolute improvement).
  • Universally Boosting Robust Losses: LogitClip acts as a force multiplier for existing robust losses (such as MAE, Generalized CE, and Focal Loss), consistently improving their performance under heavy noise.
Figure 3: Performance comparison among ReLU6, LC-V, and our method (LC-N) on noisy CIFAR-10.
  • Scalability: On the massive WebVision dataset, a ResNet-18 trained with LogitClip outperformed all other standalone robust loss functions, achieving a top-1 validation accuracy of 65.12%.

Limitations and Future Work

The primary limitation of LogitClip is its reliance on the hyperparameter $\tau$ (or its inverse $1/\tau$). The optimal clipping threshold depends heavily on the dataset, the specific architecture, and the underlying noise rate, requiring a clean validation set or careful tuning to find the sweet spot between noise immunity and underfitting.

The authors suggest that future research could focus on developing adaptive clipping mechanisms that dynamically adjust $\tau$ during training based on the model's state or estimated noise levels.

Summary updated

Abstract

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.
BibTeX
@inproceedings{wei2023mitigating,
  author = {Hongxin Wei and Huiping Zhuang and Renchunzi Xie and Lei Feng and Gang Niu and Bo An and Yixuan Li},
  title = {{Mitigating Memorization of Noisy Labels by Clipping the Model Prediction}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=g0ofsq1NRL},
}
Figure