Delving Deep into Label Smoothing

Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, Ming-Ming Cheng

Abstract

Label smoothing is a widely used regularization technique that prevents Deep Neural Networks (DNNs) from becoming over-confident by softening hard targets. However, traditional label smoothing applies a uniform distribution across all non-target classes, ignoring the inherent semantic relationships between categories. This paper introduces Online Label Smoothing (OLS), a dynamic strategy that generates soft labels based on the model's own evolving predictions. By constructing a class-level similarity matrix during training, OLS provides a more nuanced supervision signal. Experimental results across ImageNet, CIFAR-100, and various fine-grained datasets demonstrate that OLS significantly improves classification accuracy, model calibration, and robustness to both label noise and adversarial attacks.

Introduction: The Over-Confidence Problem

Deep Neural Networks are prone to overfitting, often producing highly confident predictions even when incorrect. To mitigate this, researchers employ Label Smoothing (LS), which replaces "hard" one-hot labels (e.g., [0, 0, 1, 0]) with "soft" labels that distribute a small amount of probability mass to incorrect classes.

While effective, standard LS is fundamentally limited: it treats all incorrect classes as equally likely. In reality, a "cat" is semantically closer to a "dog" than to a "truck." Standard LS ignores these structural relationships. While Knowledge Distillation (KD) can capture these nuances using a teacher model, it is computationally expensive. OLS aims to capture this "dark knowledge" without the need for an external teacher.

Methodology: Online Label Smoothing

The core idea of OLS is to let the model supervise itself by tracking its own predictive behavior. Unlike static label smoothing, OLS updates the target distribution dynamically throughout the training process.

Overview of the Online Label Smoothing (OLS) framework, showing the iterative update of class-level statistics.

The OLS process consists of three main components:

  1. Online Statistics Accumulation: During each training epoch, the model tracks its predictions for correctly classified samples. These are aggregated into a class-level matrix $S$.
  2. Class-Level Normalization: At the end of an epoch, the accumulated scores in $S$ are normalized to create a probability distribution for each category. This distribution reflects which classes the model currently perceives as similar to the target.
  3. Dynamic Supervision: In the subsequent epoch, these generated distributions serve as the soft targets. The final loss is a weighted combination of the hard cross-entropy and the loss against these OLS-generated soft labels.
Comparison of target distributions: (a) Hard Label, (b) standard Label Smoothing (LS) with uniform distribution, and (c) the proposed Online Label Smoothing (OLS) showing structured class relationships.

As shown above, OLS produces a structured distribution where semantically related classes (like 'cat' and 'dog') receive higher probability mass than unrelated ones, unlike the flat distribution of standard LS.

Key Insights and Performance

Intra-class Constraints and Feature Space

By utilizing class-level statistics, OLS encourages samples within the same category to share similar predictive distributions. t-SNE visualizations confirm that this leads to tighter, more distinct clusters in the feature space compared to standard LS, enhancing the model's discriminative power.

Robustness to Noise and Adversarial Attacks

A significant advantage of OLS is its robustness. Because the soft labels are built only from correctly predicted samples, the model effectively filters out the influence of mislabeled data. On CIFAR-100 with 60% label noise, OLS maintained a 50.04% accuracy, significantly outperforming standard LS. Furthermore, OLS-trained models showed superior resilience against adversarial attacks like FGSM, reducing error rates by over 20% compared to hard-label training.

Improved Calibration

Beyond accuracy, OLS improves Expected Calibration Error (ECE). This ensures that the model's confidence scores are more reliable indicators of actual correctness, a critical requirement for deploying DNNs in safety-critical applications.

Conclusion

Online Label Smoothing offers a simple yet powerful alternative to traditional regularization techniques. By leveraging the model's internal "dark knowledge" to build dynamic, class-aware soft labels, OLS achieves the benefits of Knowledge Distillation without the associated overhead. Its consistent performance gains across diverse benchmarks and its remarkable robustness to noise make it a valuable tool for training more reliable and accurate deep learning models.

Summary updated

Abstract

Label smoothing is an effective regularization tool for deep neural networks (DNNs), which generates soft labels by applying a weighted average between the uniform distribution and the hard label. It is often used to reduce the overfitting problem of training DNNs and further improve classification performance. In this paper, we aim to investigate how to generate more reliable soft labels. We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category. The proposed OLS constructs a more reasonable probability distribution between the target categories and non-target categories to supervise DNNs. Experiments demonstrate that based on the same classification models, the proposed approach can effectively improve the classification performance on CIFAR-100, ImageNet, and fine-grained datasets. Additionally, the proposed method can significantly improve the robustness of DNN models to noisy labels compared to current label smoothing approaches.
BibTeX
@article{zhang2021delving,
  author = {Chang-Bin Zhang and Peng-Tao Jiang and Qibin Hou and Yunchao Wei and Qi Han and Zhen Li and Ming-Ming Cheng},
  title = {{Delving Deep into Label Smoothing}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=3nxcjg0rqED},
}
Figure