Regularizing Deep Networks with Soft Target Distributions

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2494 papers645 conceptsfrozen 2026-10-04

In standard classification, we typically train models using "hard" one-hot labels. We tell the network that an image is 100% "cat" and 0% everything else. While this seems logical, it forces the model to become as confident as possible, often driving its internal representations to extremes.

Label smoothing is a simple but powerful modification to this objective. Instead of targeting a probability of 1 for the correct class, we target a slightly "smeared" distribution that reserves a small amount of probability mass for all other classes. This small change fundamentally alters how a network learns, improving its calibration, robustness, and ability to generalize.

The Problem of Infinite Logits

To understand why label smoothing works, we must look at the cross-entropy loss function. To achieve a probability of exactly 1.0 for a single class, the network's output before the softmax (the logit) for that class must theoretically reach infinity relative to the others.

As Christian Szegedy and colleagues noted in Rethinking the Inception Architecture for Computer Vision, this encourages the model to become overconfident and "brittle." If a model is forced to be 100% certain about every training example—including noisy ones—it will overfit. By replacing the hard 1 with a value like 0.9, label smoothing caps these logits. The model no longer tries to reach infinity; it settles for a "good enough" separation, which acts as a potent form of regularization.

The Geometry of Representations

What happens inside the model when we smooth the labels? Rafael Müller, Geoffrey E. Hinton, and Simon Kornblith visualized this in When does label smoothing help?.

They found that label smoothing forces the penultimate layer's representations to form tight, equidistant clusters. In a hard-labeled model, a "cat" that looks slightly like a "dog" might be pushed toward the dog cluster in feature space. Label smoothing, however, encourages every class to be equally distant from every other class. This geometric regularity is why label-smoothed models are so well-calibrated: their internal confidence scores actually match their real-world accuracy.

A Shield Against Noise and Forgetting

Because label smoothing prevents the model from "locking in" on specific labels with absolute certainty, it provides a natural defense against errors in the data.

  1. Noisy Labels: In Rethinking Label Refurbishment, Yangdi Lu and colleagues show that "Extreme Label Smoothing"—where the target is nearly uniform—prevents the model from memorizing noisy outliers. It effectively slows down the learning process, ensuring the model only captures the most persistent patterns.
  2. Catastrophic Forgetting: When fine-tuning Large Language Models for new tasks, like adding visual capabilities, the model often "forgets" its original linguistic skills. Shikhar Srivastava demonstrated in Improving Multimodal Large Language Models Using Continual Learning that using soft targets prevents the model from over-optimizing for the new data, preserving its original knowledge.
  3. Adversarial Robustness: By smoothing the optimization landscape, label smoothing makes it harder for small perturbations to flip a model's prediction. Sanghyuk Chun and others found in their Empirical Evaluation that this lack of overconfidence is a key ingredient in building models that resist adversarial attacks.

Domain-Specific Smoothing

The "uniform" smoothing introduced by Szegedy is just the beginning. Researchers have found that the shape of the smoothing should often match the structure of the problem:

  • Spatial Smoothing: In trajectory prediction, Viktor Kress used Gaussian-shaped smoothing in Pose and Semantic Map Based Probabilistic Forecast. Since a pedestrian's future location is likely to be near the ground truth, smoothing the label across adjacent spatial grid cells creates a more realistic training signal.
  • Semantic Smoothing: In vision-language tasks, Youngtaek Oh and colleagues used a "gentler" regularization in FSC-CLIP by assigning small positive values to "hard negatives"—samples that are technically incorrect but semantically very similar to the truth.

The Distillation Trade-off

Despite its benefits, label smoothing has a notable downside: it makes for a poor teacher. As Müller et al. discovered, because label smoothing erases the nuanced similarities between classes (by forcing them into equidistant clusters), a student model learning from a smoothed teacher loses the "dark knowledge" that usually makes knowledge distillation effective. If you want a model to be accurate and calibrated, smooth its labels; if you want it to teach another model, stick to hard targets.

Papers in this version

Showing 60 of 494 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 458 papers
  2. Version 1 · 2026-10-04 · 494 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_label_smoothing_942ffca1,
  title = {label smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/label-smoothing/rdr_942ffca1474c45c486d6859a51fff820},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_regularizing_deep_networks_with_soft_tar_942ffca1_v1,
  title = {Regularizing Deep Networks with Soft Target Distributions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/regularizing-deep-networks-with-soft-target-distributions/rdr_942ffca1474c45c486d6859a51fff820/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}