TL;DR
Standard Cross Entropy (CE) loss forces deep neural networks to memorize incorrect labels because its mathematical formulation is unbounded. By simply clipping the norm of the model's output logits (LogitClip), this paper introduces a universal, plug-and-play constraint that prevents networks from overfitting to label noise while consistently boosting generalization performance across various architectures and datasets.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are notoriously good at memorizing things—including mistakes. When trained on datasets containing "noisy" (incorrect) labels, standard Cross Entropy (CE) loss forces the model to confidently predict these wrong labels. Why? Because CE loss is unbounded; if the model disagrees with a training label, the loss approaches infinity. To avoid this massive penalty, the model warps its decision boundaries to overfit the noise, severely damaging its ability to generalize to clean test data.
While researchers have designed specialized "robust" loss functions to combat this, these alternatives often introduce optimization difficulties, leading to underfitting on complex datasets. This paper tackles a fundamental question: Can we make existing loss functions noise-robust without changing the loss functions themselves?
Methods
Instead of altering the loss function, the authors propose Logit Clipping (LogitClip), which bounds the loss at the logit level (the raw, unnormalized outputs of the model before the softmax layer).
Mathematically, LogitClip clamps the $\ell_p$-norm of the logit vector $z$ to a maximum threshold $\tau$:
$$ \text{clip}_\tau(z) = \begin{cases} \tau \cdot \frac{z}{\|z\|_p} & \text{if } \|z\|_p \ge \tau \\ z & \text{else} \end{cases} $$
Why Clip by Norm?
Unlike standard value clipping (e.g., clamping individual logit values between $[-6, 6]$), clipping the norm preserves the relative proportions and direction of the logit vector. This means the model's final class prediction remains completely unchanged, but its maximum possible loss is strictly capped.
The authors provide theoretical proofs showing that equipping CE loss with LogitClip guarantees strict upper and lower bounds on the loss, effectively preventing the gradient from exploding when encountering a noisy label.
Datasets
The authors evaluated LogitClip across a comprehensive suite of benchmarks with both simulated and real-world label noise:
- CIFAR-10 & CIFAR-100: Evaluated using three simulated noise types: Symmetric (random flipping), Asymmetric (flipping to similar classes), and Instance-dependent (noise based on image features).
- CIFAR-10N & CIFAR-100N: Human-annotated real-world noisy versions of the CIFAR datasets.
- WebVision: A large-scale, real-world noisy dataset containing 2.4 million web images.
Key Insights and Ideas
- Logit-Level vs. Loss-Level Modification: Instead of engineering complex new loss functions, controlling the magnitude of the network's raw outputs is a simpler, more direct way to enforce loss boundedness.
- The Direction Matters: Value-clipping methods (like ReLU6) alter the direction of the logit vector, which can inadvertently change the model's predictions and zero out gradients. Norm-clipping maintains the model's predictive intent while safely managing its confidence.
- The Goldilocks Zone of $\tau$: The hyperparameter $\tau$ controls the noise-tolerance threshold. As shown in Figure 2, a smaller $\tau$ (larger $1/\tau$) yields tighter theoretical risk bounds against noise. However, if $\tau$ is too small, the loss bounds collapse together, making optimization difficult and causing the model to underfit even clean data.
Key Results
LogitClip achieved massive performance gains across the board, proving to be both model-agnostic and loss-agnostic.
- Revitalizing Cross Entropy: On CIFAR-10 with instance-dependent noise, adding LogitClip skyrocketed the test accuracy of standard CE loss from 68.36% to 86.74% (an 18.38% absolute improvement).
- Universally Boosting Robust Losses: LogitClip acts as a force multiplier for existing robust losses (such as MAE, Generalized CE, and Focal Loss), consistently improving their performance under heavy noise.
- Scalability: On the massive WebVision dataset, a ResNet-18 trained with LogitClip outperformed all other standalone robust loss functions, achieving a top-1 validation accuracy of 65.12%.
Limitations and Future Work
The primary limitation of LogitClip is its reliance on the hyperparameter $\tau$ (or its inverse $1/\tau$). The optimal clipping threshold depends heavily on the dataset, the specific architecture, and the underlying noise rate, requiring a clean validation set or careful tuning to find the sweet spot between noise immunity and underfitting.
The authors suggest that future research could focus on developing adaptive clipping mechanisms that dynamically adjust $\tau$ during training based on the model's state or estimated noise levels.
Summary updated