Introduction
Deep neural networks are notoriously sensitive to label noise. In large-scale datasets, images are frequently mislabeled—for instance, a cat might be incorrectly tagged as a dog. Standard training typically relies on Cross Entropy (CE) loss, which is mathematically unbounded. When a model encounters a noisy label that contradicts its high-confidence prediction, the resulting loss can approach infinity. This forces the network to "memorize" the error to minimize the loss, severely compromising its ability to generalize to clean data.
The Logit Clipping (LogitClip) Method
To address this, researchers have introduced Logit Clipping (LogitClip). Rather than redesigning the loss function, LogitClip intervenes at the logit level—the raw scores produced by the model before the final softmax layer. The technique ensures that the norm of the logit vector $z$ never exceeds a predefined threshold $\tau$. If the norm $\|z\|_p$ surpasses $\tau$, the vector is scaled down:
$$clip_\tau(z) = \tau \cdot \frac{z}{\|z\|_p} \text{ if } \|z\|_p \ge \tau, \text{ else } z$$
Why Logit Clipping Works
By capping the magnitude of the logits, LogitClip effectively places a ceiling on the maximum possible loss. This prevents the "infinite" penalties that drive noise memorization. Crucially, because the entire vector is scaled, the relative ranking of classes is preserved, maintaining the model's intended prediction direction.
Experimental Evaluation
The effectiveness of LogitClip was evaluated across several benchmarks, including CIFAR-10 and CIFAR-100 with simulated noise (symmetric, asymmetric, and instance-dependent), as well as real-world noisy datasets like CIFAR-10N, CIFAR-100N, and the massive WebVision dataset (2.4 million images).
Key Results and Universal Enhancement
One of the most significant findings is that LogitClip acts as a "universal enhancer." It is not limited to CE loss; it can be integrated with nearly any existing robust loss function, such as Focal loss, MAE, or Generalized Cross Entropy (GCE).
As shown in the results, LogitClip provides massive accuracy boosts. For example, on CIFAR-10 with instance-dependent noise, it improved the test accuracy of standard CE loss from approximately 68% to over 86%. The method also achieved state-of-the-art results on the WebVision "Mini" setting when combined with other advanced techniques.
Conclusion and Future Work
LogitClip offers a simple, plug-and-play solution to the problem of noise memorization by addressing the unbounded nature of traditional loss functions. While it requires tuning the hyperparameter $\tau$, its ability to improve a wide range of existing methods makes it a powerful tool for robust deep learning. Future research may explore how this logit-level intervention interacts with other training components like data augmentation and learning rate schedules.
Summary updated