Logit Clipping for Robust Learning against Label Noise

Hongxin Wei, Huiping Zhuang, Renchunzi Xie, Lei Feng, Gang Niu, Bo An, Yixuan Li

Introduction

Deep neural networks are notoriously sensitive to label noise. In large-scale datasets, images are frequently mislabeled—for instance, a cat might be incorrectly tagged as a dog. Standard training typically relies on Cross Entropy (CE) loss, which is mathematically unbounded. When a model encounters a noisy label that contradicts its high-confidence prediction, the resulting loss can approach infinity. This forces the network to "memorize" the error to minimize the loss, severely compromising its ability to generalize to clean data.

The Logit Clipping (LogitClip) Method

To address this, researchers have introduced Logit Clipping (LogitClip). Rather than redesigning the loss function, LogitClip intervenes at the logit level—the raw scores produced by the model before the final softmax layer. The technique ensures that the norm of the logit vector $z$ never exceeds a predefined threshold $\tau$. If the norm $\|z\|_p$ surpasses $\tau$, the vector is scaled down:

$$clip_\tau(z) = \tau \cdot \frac{z}{\|z\|_p} \text{ if } \|z\|_p \ge \tau, \text{ else } z$$

Why Logit Clipping Works

By capping the magnitude of the logits, LogitClip effectively places a ceiling on the maximum possible loss. This prevents the "infinite" penalties that drive noise memorization. Crucially, because the entire vector is scaled, the relative ranking of classes is preserved, maintaining the model's intended prediction direction.

Figure 1: Theoretical loss bounds as a function of the clipping threshold $\tau$. Capping the logits prevents the loss from becoming unbounded, mitigating the pressure to overfit to noisy labels.

Experimental Evaluation

The effectiveness of LogitClip was evaluated across several benchmarks, including CIFAR-10 and CIFAR-100 with simulated noise (symmetric, asymmetric, and instance-dependent), as well as real-world noisy datasets like CIFAR-10N, CIFAR-100N, and the massive WebVision dataset (2.4 million images).

Key Results and Universal Enhancement

One of the most significant findings is that LogitClip acts as a "universal enhancer." It is not limited to CE loss; it can be integrated with nearly any existing robust loss function, such as Focal loss, MAE, or Generalized Cross Entropy (GCE).

Table 1: Performance comparison across various loss functions with and without Logit Clipping (+LC). The addition of LogitClip consistently improves test accuracy across different noise types and base methods.

As shown in the results, LogitClip provides massive accuracy boosts. For example, on CIFAR-10 with instance-dependent noise, it improved the test accuracy of standard CE loss from approximately 68% to over 86%. The method also achieved state-of-the-art results on the WebVision "Mini" setting when combined with other advanced techniques.

Conclusion and Future Work

LogitClip offers a simple, plug-and-play solution to the problem of noise memorization by addressing the unbounded nature of traditional loss functions. While it requires tuning the hyperparameter $\tau$, its ability to improve a wide range of existing methods makes it a powerful tool for robust deep learning. Future research may explore how this logit-level intervention interacts with other training components like data augmentation and learning rate schedules.

Summary updated

BibTeX
@inproceedings{wei2023logit,
  author = {Hongxin Wei and Huiping Zhuang and Renchunzi Xie and Lei Feng and Gang Niu and Bo An and Yixuan Li},
  title = {{Logit Clipping for Robust Learning against Label Noise}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=g0ofsq1NRL},
}
Figure