Abstract
Label smoothing is a widely used regularization technique that prevents Deep Neural Networks (DNNs) from becoming over-confident by softening hard targets. However, traditional label smoothing applies a uniform distribution across all non-target classes, ignoring the inherent semantic relationships between categories. This paper introduces Online Label Smoothing (OLS), a dynamic strategy that generates soft labels based on the model's own evolving predictions. By constructing a class-level similarity matrix during training, OLS provides a more nuanced supervision signal. Experimental results across ImageNet, CIFAR-100, and various fine-grained datasets demonstrate that OLS significantly improves classification accuracy, model calibration, and robustness to both label noise and adversarial attacks.
Introduction: The Over-Confidence Problem
Deep Neural Networks are prone to overfitting, often producing highly confident predictions even when incorrect. To mitigate this, researchers employ Label Smoothing (LS), which replaces "hard" one-hot labels (e.g., [0, 0, 1, 0]) with "soft" labels that distribute a small amount of probability mass to incorrect classes.
While effective, standard LS is fundamentally limited: it treats all incorrect classes as equally likely. In reality, a "cat" is semantically closer to a "dog" than to a "truck." Standard LS ignores these structural relationships. While Knowledge Distillation (KD) can capture these nuances using a teacher model, it is computationally expensive. OLS aims to capture this "dark knowledge" without the need for an external teacher.
Methodology: Online Label Smoothing
The core idea of OLS is to let the model supervise itself by tracking its own predictive behavior. Unlike static label smoothing, OLS updates the target distribution dynamically throughout the training process.
The OLS process consists of three main components:
- Online Statistics Accumulation: During each training epoch, the model tracks its predictions for correctly classified samples. These are aggregated into a class-level matrix $S$.
- Class-Level Normalization: At the end of an epoch, the accumulated scores in $S$ are normalized to create a probability distribution for each category. This distribution reflects which classes the model currently perceives as similar to the target.
- Dynamic Supervision: In the subsequent epoch, these generated distributions serve as the soft targets. The final loss is a weighted combination of the hard cross-entropy and the loss against these OLS-generated soft labels.
As shown above, OLS produces a structured distribution where semantically related classes (like 'cat' and 'dog') receive higher probability mass than unrelated ones, unlike the flat distribution of standard LS.
Key Insights and Performance
Intra-class Constraints and Feature Space
By utilizing class-level statistics, OLS encourages samples within the same category to share similar predictive distributions. t-SNE visualizations confirm that this leads to tighter, more distinct clusters in the feature space compared to standard LS, enhancing the model's discriminative power.
Robustness to Noise and Adversarial Attacks
A significant advantage of OLS is its robustness. Because the soft labels are built only from correctly predicted samples, the model effectively filters out the influence of mislabeled data. On CIFAR-100 with 60% label noise, OLS maintained a 50.04% accuracy, significantly outperforming standard LS. Furthermore, OLS-trained models showed superior resilience against adversarial attacks like FGSM, reducing error rates by over 20% compared to hard-label training.
Improved Calibration
Beyond accuracy, OLS improves Expected Calibration Error (ECE). This ensures that the model's confidence scores are more reliable indicators of actual correctness, a critical requirement for deploying DNNs in safety-critical applications.
Conclusion
Online Label Smoothing offers a simple yet powerful alternative to traditional regularization techniques. By leveraging the model's internal "dark knowledge" to build dynamic, class-aware soft labels, OLS achieves the benefits of Knowledge Distillation without the associated overhead. Its consistent performance gains across diverse benchmarks and its remarkable robustness to noise make it a valuable tool for training more reliable and accurate deep learning models.
Summary updated