Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It

Guoxuan Xia, Olivier Laurent, Gianni Franchi, Christos-Savvas Bouganis

In the quest for higher accuracy, deep learning practitioners often reach for Label Smoothing (LS). By softening "hard" one-hot targets, LS acts as a powerful regularizer that prevents overfitting and often boosts top-1 accuracy. However, a new paper from ICLR 2025 reveals a hidden cost: LS significantly impairs a model's ability to perform Selective Classification (SC)—the crucial task of knowing when to "abstain" or "reject" a prediction due to high uncertainty.

TL;DR

This paper demonstrates that Label Smoothing consistently degrades selective classification performance across various architectures and tasks. The authors provide a novel gradient-level analysis showing that LS suppresses the maximum logit more for correct predictions than for incorrect ones, ruining uncertainty rankings; they also show that post-hoc logit normalization can effectively recover this lost performance.

What Problem Does This Paper Address?

In safety-critical applications like autonomous driving or medical diagnosis, a model shouldn't just be accurate; it needs to be self-aware. Selective Classification (SC) allows a model to reject a prediction if its uncertainty exceeds a certain threshold. The goal is to minimize risk (error rate on accepted samples) while maximizing coverage (the percentage of samples the model actually predicts).

While LS is a staple in modern training recipes (like those for Vision Transformers), researchers have noticed that these high-accuracy models often perform surprisingly poorly at SC. They are "overconfident" in their errors. This paper sets out to empirically prove this degradation is widespread and, more importantly, to explain the mathematical "why" behind it.

Label Smoothing vs. Selective Classification concepts

Methods

The authors tackle the problem through three main avenues:

  1. Extensive Empirical Evaluation: They test LS across a wide range of architectures (ResNet, ViT, DenseNet, MLP, LSTM) and modalities (ImageNet classification, Cityscapes segmentation, tabular data, and NLP).
  2. Logit-Level Gradient Analysis: This is the core theoretical contribution. By comparing the gradients of the LS loss vs. standard Cross-Entropy (CE), they analyze how LS influences the training of the final layer's outputs (logits).
  3. Post-hoc Mitigation: They evaluate logit normalization—dividing the logit vector by its $p$-norm during inference—as a way to fix the ranking issues caused by LS.

Datasets

The study uses a diverse set of benchmarks to ensure generalizability:

  • ImageNet-1k: The standard large-scale image classification benchmark.
  • Cityscapes: For pixel-level semantic segmentation.
  • ImageNet-Sketch: Used to introduce artificial distribution shift and test how LS handles "out-of-distribution" uncertainty.
  • CIFAR-100: For small-scale image classification.
  • Bank Marketing & Online Shoppers: For tabular binary classification.
  • IMDB Movie Review: For natural language processing (sentiment analysis).

Key Insights and Ideas

  • The "Aha" Moment: Imbalanced Suppression. The authors show that the amount LS suppresses the maximum logit ($v_{max}$) is directly tied to the true probability of error. Crucially, LS suppresses $v_{max}$ more when a prediction is likely to be correct and less when it is likely to be wrong. This "imbalanced suppression" narrows the gap between correct and incorrect predictions, making them harder to distinguish via uncertainty scores like Maximum Softmax Probability (MSP).
  • Calibration $\neq$ Selective Classification. LS is known to improve model calibration (aligning confidence with accuracy). However, the authors emphasize that SC depends on the relative ranking of uncertainties, not their absolute values. A model can be perfectly calibrated but still have a poor uncertainty ranking that hurts SC.
  • Logit Norm as a Compensator. Logit normalization works because it increases uncertainty more for samples with higher $v_{max}$. Since LS-trained models end up with higher $v_{max}$ for errors (relative to correct samples) compared to CE models, the $p$-norm normalization effectively "penalizes" these overconfident errors, restoring a better ranking.
Distribution of max logits for correct vs incorrect predictions

Key Results

  • Consistent Degradation: Across all tasks, LS models showed worse Risk-Coverage (RC) curves than CE models. As the smoothing factor $\alpha$ increases, SC performance drops further.
  • The 1% Risk Failure: On ImageNet, if you require a strict 1% error rate on accepted samples, LS models often achieve near-zero coverage, meaning they can't confidently accept any samples at that safety level. In contrast, CE models can accept ~15% of the data.
  • Recovery via Normalization: Applying logit normalization to LS models significantly recovers SC performance. In some cases, LS + Logit Norm even outperformed vanilla CE (e.g., reaching 22.31% coverage at 1% risk on ImageNet).
  • Visual Evidence in Segmentation: In semantic segmentation (Cityscapes), LS models are shown to be highly confident (low uncertainty rank) in regions where they have clearly misclassified objects (e.g., predicting a sidewalk as a road). Logit normalization visibly mitigates these overconfident errors.
Risk-Coverage curves showing LS degradation across models

Limitations and Future Work

  • Hyperparameter Tuning: Logit normalization requires a validation set to find the optimal $p$ value (typically via a grid search for the lowest Area Under the Risk-Coverage curve).
  • Not a Total Cure: While effective, logit normalization doesn't fix every error. In segmentation, some boundary-related errors remain highly confident even after normalization.
  • Beyond LS: The authors suggest that other label-altering techniques, like MixUp, might suffer from similar issues and warrant a similar gradient-based investigation.
  • Generalization of Analysis: The gradient analysis assumes the model is not perfectly fit to all data regions. Future work could explore how this interacts with epistemic uncertainty and out-of-distribution detection.

Summary updated

Abstract

Label smoothing (LS) is a popular regularisation method for training neural networks as it is effective in improving test accuracy and is simple to implement. ``Hard'' one-hot labels are ``smoothed'' by uniformly distributing probability mass to other classes, reducing overfitting. Prior work has suggested that in some cases LS can degrade selective classification (SC) -- where the aim is to reject misclassifications using a model's uncertainty. In this work, we first demonstrate empirically across an extended range of large-scale tasks and architectures that LS consistently degrades SC. We then address a gap in existing knowledge, providing an explanation for this behaviour by analysing logit-level gradients: LS degrades the uncertainty rank ordering of correct vs incorrect predictions by suppressing the max logit more when a prediction is likely to be correct, and less when it is likely to be wrong. This elucidates previously reported experimental results where strong classifiers underperform in SC. We then demonstrate the empirical effectiveness of post-hoc logit normalisation for recovering lost SC performance caused by LS. Furthermore, linking back to our gradient analysis, we again provide an explanation for why such normalisation is effective.
BibTeX
@inproceedings{xia2025towards,
  author = {Guoxuan Xia and Olivier Laurent and Gianni Franchi and Christos-Savvas Bouganis},
  title = {{Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=6oWFn6fY4A},
}
Figure