Regularizing Neural Networks via Label Smoothing

663 concepts430 papers2 proposals

In deep learning, classification models are typically trained using the cross-entropy loss paired with one-hot encoded target vectors. Under this paradigm, a model is given a binary directive: assign a probability of exactly $1.0$ to the correct class and $0.0$ to all others.

To achieve a loss of zero under a softmax activation function, the network must drive the logit (the unnormalized network output) of the correct class toward positive infinity relative to the incorrect classes. This mathematical imperative forces networks to become highly overconfident in their predictions. When training data contains noise, ambiguous boundaries, or limited samples, this pursuit of absolute certainty leads directly to overfitting and poor calibration.

Label smoothing is a simple and elegant regularization technique that addresses this fundamental flaw by replacing hard, unyielding targets with soft, probabilistic distributions.

The Mechanics of Soft Targets

Popularized by Christian Szegedy et al. in Rethinking the Inception Architecture for Computer Vision, label smoothing modifies the target distribution by taking a weighted average of the original one-hot ground truth and a uniform distribution across all classes.

If a dataset contains $K$ classes and a smoothing parameter $\epsilon$, the new target $y_{ls}$ for a true class $k$ is defined as:

$$y_{ls}(i) = \begin{cases} 1 - \epsilon + \frac{\epsilon}{K} & \text{if } i = k \\ \frac{\epsilon}{K} & \text{if } i \neq k \end{cases}$$

By allocating a small fraction of probability mass ($\epsilon$) to the incorrect classes, the network is no longer penalized for remaining slightly uncertain. The optimal logit for the correct class is capped at a finite value rather than being pushed toward infinity. This simple adjustment fundamentally alters the optimization landscape and the internal geometry of the network.

The Geometry of the Representation Space

Why does softening labels improve a network's ability to generalize? The answer lies in how the network structures its internal representations.

In When does label smoothing help?, Rafael Müller et al. analyzed the activations of the penultimate layer of deep neural networks. When trained with hard one-hot labels, the representations within a single class tend to be highly elongated and spread out. Because the optimization objective continuously demands larger logits, the model forces examples of the same class apart to maximize the distance to the decision boundary.

Hard Labels:
   [ Class A Representation ]  ----> (Pushed infinitely apart to maximize logits)

Label Smoothing:
   ( Equidistant, tight clusters centered around a shared prototype )

When label smoothing is introduced, the objective changes. The model is incentivized to ensure that the logit of the correct class maintains a fixed, finite difference from the average of the incorrect logits. Geometrically, this forces the penultimate layer representations to collapse into tight, equidistant clusters centered around a shared prototype.

This clustering behavior prevents gradient saturation. As Kien Do et al. noted in Clustering by Maximizing Mutual Information Across Views, preventing cluster assignments from becoming prematurely sharp ensures that gradients remain informative, allowing continuous learning throughout optimization. Furthermore, Michal Lukasik demonstrated in Does label smoothing mitigate label noise? that in linear models, this mechanism acts as a form of shrinkage regularization. By encouraging smaller model weights, label smoothing increases the classification margin and prevents the decision boundary from overfitting to outliers.

Aligning Confidence with Reality

A model is considered well-calibrated if its predicted probabilities match its empirical likelihood of being correct. If a calibrated model predicts a diagnosis with 90% confidence, it should be correct exactly 90% of the time. Standard deep networks are notoriously miscalibrated, frequently outputting near-100% confidence even when making incorrect predictions.

Because label smoothing caps the maximum target probability, it serves as an implicit calibration tool. The authors of When does label smoothing help? demonstrated that it dramatically reduces expected calibration error (ECE) without requiring post-hoc adjustments like temperature scaling.

This property makes label smoothing indispensable in high-stakes domains where uncertainty quantification is critical:

A Buffer Against Noisy Supervision

Real-world datasets—especially those scraped from the web—are riddled with mislabeled instances, ambiguous examples, and false negatives. Hard labels force a model to memorize these errors, corrupting its feature representations. Label smoothing acts as a robust safety net in these scenarios.

Mitigating False Negatives in Contrastive Learning

In large-scale vision-language pre-training (like CLIP), models learn by assuming that an image and its paired caption are the only positive match in a batch, treating all other captions as absolute negatives. However, web data often contains multiple images that match a single description. Enrico Fini et al. demonstrated in Improved baselines for vision-language pre-training that applying label smoothing to contrastive objectives distributes probability mass across negative samples, mitigating the harsh penalties associated with unobserved false negatives. This concept is mirrored in multi-label learning from single positive labels, where Elijah Cole et al. in Multi-Label Learning From Single Positive Labels used label smoothing to absorb the noise of missing annotations.

Stabilizing Pseudo-Labeling Loops

In semi-supervised and partial label learning, models often generate their own "pseudo-labels" to train on unlabeled data. If a model becomes overconfident in an incorrect self-generated label, it enters a destructive feedback loop. Darshana Saravanan et al. addressed this in Pseudo-labelling meets Label Smoothing for Noisy Partial Label Learning, showing that softening targets prevents the classifier from over-indexing on potentially erroneous pseudo-labels, maintaining model robustness even in high-noise environments.

Handling the Long Tail

When dealing with instance-level label noise, the risks are unevenly distributed. Yang Liu et al. in Understanding Instance-Level Label Noise: Disparate Impacts and Treatments observed that aggressive loss-correction techniques often fail on rare, long-tail data points because they mistakenly flip labels based on scarce evidence. Label smoothing offers a conservative, reliable alternative by reducing model overconfidence across the board without making brittle assumptions about the underlying noise distribution.

Advanced Variants: Negative and Adaptive Smoothing

While uniform label smoothing is highly effective when noise is distributed evenly, it can falter in extreme scenarios. In heavily corrupted datasets, redistributing uniform probability mass to non-target classes can "over-smooth" the model, flattening the estimated posterior distribution to the point where the model becomes under-confident.

To combat this, Jiaheng Wei et al. proposed Negative Label Smoothing (NLS) in To Smooth or Not? When Label Smoothing Meets Noisy Labels. Instead of adding probability mass to non-target classes, NLS subtracts it by applying a negative weight parameter.

Standard Label Smoothing:
   [True Class: 0.9]  --> [Incorrect Class 1: 0.05]  --> [Incorrect Class 2: 0.05]

Negative Label Smoothing:
   [True Class: 1.1]  --> [Incorrect Class 1: -0.05] --> [Incorrect Class 2: -0.05]

This mechanism penalizes the model for assigning probability to non-target classes, effectively sharpening the decision boundaries. In high-noise regimes, this negative weight acts as a corrective force that cancels out the entropy introduced by symmetric label corruption, preventing the posterior from becoming too flat.

Taking this a step further, Min Wang et al. developed a unified framework in Moderate Message Passing Improves Calibration: A Universal Way to Mitigate Confidence Bias in Graph Neural Networks that dynamically integrates both label softening and sharpening into a single objective. By adjusting the target distribution's entropy based on the model's real-time performance, it simultaneously corrects for both over-confident and under-confident predictions.

The Hidden Trade-offs

Despite its widespread utility, label smoothing introduces subtle costs that practitioners must navigate.

The Distillation Paradox

The very geometric property that makes label smoothing an effective regularizer—collapsing representations into tight, equidistant clusters—makes it a poor choice if the model is intended to be a teacher for knowledge distillation. As discovered by Rafael Müller et al., by forcing all incorrect classes to be treated uniformly, label smoothing erases the nuanced, fine-grained similarities between classes (e.g., ensuring a picture of a truck is recognized as more similar to a car than to an apple). When a student model attempts to learn from a smoothed teacher, it loses access to this rich, structural dark knowledge.

Gradient Masking and Security Illusions

In the realm of adversarial robustness, label smoothing can create a dangerous illusion of security. Xingjun Ma demonstrated in Imbalanced gradients: a subtle cause of overestimated adversarial robustness that label smoothing can cause an "imbalanced gradient" phenomenon. By suppressing the gradients of incorrect class logits, the gradient of the correct class dominates the optimization landscape. This effectively shields the model from standard gradient-based adversarial attacks during evaluation, masking the model's true underlying vulnerability without actually making it more secure against sophisticated, margin-decomposed attacks.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 458 papers
  2. Version 1 · 2026-10-04 · 494 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_label_smoothing_942ffca1,
  title = {label smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-smoothing/rdr_942ffca1474c45c486d6859a51fff820},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_label_smoothing_942ffca1_v2,
  title = {label smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-smoothing/rdr_942ffca1474c45c486d6859a51fff820/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake