TL;DR
While Label Smoothing (LS) is a go-to trick for boosting neural network accuracy, it has a hidden cost: it severely degrades Selective Classification, making models overconfident when they are actually wrong. This paper identifies the root cause—an imbalanced suppression of logit gradients—and demonstrates that a simple post-hoc "logit normalization" can recover the lost reliability.
Introduction: The Hidden Cost of Label Smoothing
In safety-critical applications like autonomous driving or medical diagnosis, we don't just need a model to be accurate; we need it to know when it’s likely to fail. This is the domain of Selective Classification (SC), where a model is given a "reject" option to abstain from predictions when its uncertainty is too high.
Practitioners often use Label Smoothing (LS) during training to prevent overfitting. Instead of training on "hard" one-hot labels (e.g., [0, 1, 0]), LS uses "soft" targets (e.g., [0.05, 0.9, 0.05]). While this improves general accuracy, researchers noticed it often makes models worse at SC. This paper sets out to prove this degradation happens consistently across tasks and, more importantly, to explain why it happens at a mathematical level.
Methods: Benchmarking and Gradient Analysis
The authors tackle the problem through a combination of large-scale empirical testing and gradient analysis:
- Empirical Benchmarking: They evaluated various architectures (ResNet, ViT, DenseNet) across multiple modalities (Images, Text, Tabular data) to confirm that LS consistently hurts SC performance. The study is remarkably broad, using standard benchmarks like ImageNet-1k, Cityscapes, ImageNet-Sketch, CIFAR-100, Bank Marketing, Online Shoppers, and IMDB Movie Reviews to ensure the findings aren't specific to one niche.
- Gradient Analysis: The core of the paper is an analysis of logit-level gradients. They compared the gradients of standard Cross-Entropy (CE) loss against LS loss. By defining a "suppression gradient," they visualized how LS pushes down on the maximum logit during training compared to standard CE.
- Post-hoc Logit Normalization: To fix the issue, they applied a deployment-time technique where logits are normalized by their $p$-norm before calculating the softmax probability. This doesn't require retraining the model; it's a simple mathematical transformation applied during inference.
Key Insights: Imbalanced Logit Suppression
The "Aha!" moment comes from the gradient analysis. The authors show that LS suppresses the maximum logit (the model's confidence) more when a prediction is likely to be correct and less when it is likely to be wrong.
While LS is known to improve calibration (making the probability match the actual frequency of correctness), SC relies on the relative ranking of uncertainty. By squashing the logits of correct predictions more aggressively than incorrect ones, LS ruins the model's ability to distinguish between "I'm sure" and "I'm guessing."
In low-risk regimes (e.g., wanting only a 1% error rate), LS-trained models often have near-zero "coverage," meaning they can't find any samples they are confident enough to accept under strict safety constraints. This creates a significant "safety gap."
Logit normalization works because it effectively increases uncertainty more for samples with higher max logits. Since LS disproportionately suppressed those "correct-looking" samples, the normalization helps restore the proper rank ordering.
Key Results and Visual Evidence
Across all tested tasks, increasing the smoothing factor ($\alpha$) led to worse SC performance. On ImageNet, at a strict 1% risk tolerance, CE models could accept ~15.66% of samples, while LS models accepted nearly 0%. Applying logit normalization recovered nearly all the lost SC performance. In many cases, the "LS + Logit Norm" combination yielded the best of both worlds: the high accuracy of label smoothing with the reliable uncertainty of standard training.
In semantic segmentation tasks like Cityscapes, LS models were shown to be highly overconfident in erroneous regions (e.g., mislabeling a sidewalk as a road). As seen in the uncertainty ranking maps, the LS model fails to flag these errors with high uncertainty compared to the standard Cross-Entropy model. Logit normalization successfully restores the ability to flag these regions as high-uncertainty.
Limitations and Future Work
- The "Positive Logit" Assumption: The mathematical proof for why logit normalization works assumes that the largest logits are positive. While the authors show this is almost always true in practice, it is a theoretical constraint.
- Other Regularizers: The paper focuses on Label Smoothing. However, other popular techniques like MixUp or CutMix also modify labels. The authors suggest that these might have similar negative effects on SC and deserve their own gradient-level investigation.
- Beyond Softmax: The study focuses on softmax-based uncertainty (Maximum Softmax Probability). While this is the most common metric, exploring how LS affects more complex uncertainty estimators (like Ensembles or Bayesian methods) remains an open question.
Summary updated