Extending Label Smoothing Regularization with Self-Knowledge Distillation

Jiyue Wang, Pei Zhang, Wenfeng Pang, Jie LiarXiv 2020 · DOI · arXiv

Abstract

Inspired by the strong correlation between Label Smoothing Regularization (LSR) and Knowledge Distillation (KD), we propose LsrKD, an algorithm that extends LSR into the KD regime by applying a softer temperature. We further enhance this with Teacher Correction (TC), which manually adjusts the teacher distribution to favor the correct class. To push performance further, we introduce Memory-replay Knowledge Distillation (MrKD), a self-distillation method that uses historical model snapshots as teachers. MrKD penalizes the divergence between current outputs and previous iterations, stabilizing training and finding more robust minima. Our experiments demonstrate that LsrKD and MrKD consistently outperform standard LSR across various deep neural networks, even in cases where LSR fails to provide benefits.

Introduction

In the quest to train more robust deep learning models, researchers often turn to two popular techniques: Label Smoothing Regularization (LSR) and Knowledge Distillation (KD). LSR prevents overconfidence by softening "one-hot" labels with a uniform distribution, while KD transfers "dark knowledge" from a complex teacher model to a simpler student. While they appear distinct, they are two sides of the same coin: both provide a more informative target distribution than hard labels.

This paper bridges these concepts by reformulating LSR as a special case of KD. We introduce LsrKD and its evolution, Memory-replay Knowledge Distillation (MrKD). These methods leverage a model's own training trajectory to provide a knowledgeable teacher without the computational overhead of pre-training an external model.

Methodology

1. LsrKD: Softening the Uniform Teacher

Standard LSR can be viewed as KD where the teacher is a fixed uniform distribution. However, LSR typically uses a "hard" softmax. We propose LsrKD, which introduces a temperature parameter ($\tau$) to the softmax function. By increasing $\tau$, the uniform distribution becomes "softer," revealing more structural information that acts as a more effective regularizer than standard LSR.

2. Teacher Correction (TC)

A purely uniform teacher can be "lifeless," treating all classes equally regardless of the true label. Teacher Correction (TC) addresses this by manually setting a slightly higher constant probability for the correct class within the teacher distribution. This provides a more reliable signal while maintaining the benefits of smoothing.

3. Memory-replay Knowledge Distillation (MrKD)

The most significant contribution is MrKD. Instead of a static uniform distribution, MrKD uses historical backups of the model itself as the teacher. By saving model weights every $\kappa$ epochs, we create a "memory" of the model's past. The current model is then trained to match the output distributions of these historical versions.

Comparison of output distributions for different regularization methods. Top row: Teacher distributions. Bottom row: Student distributions.

As shown in Figure 1, MrKD (d, e) provides a much richer teacher distribution compared to the uniform approach of LSR (a). This "dark knowledge" captures the relationships between classes as learned by the model earlier in its training trajectory.

Experimental Results

We evaluated our methods on CIFAR-10, CIFAR-100, and CINIC-10 using various architectures, including ResNet, WideResNet (WRN), and ResNeXt.

Performance on CIFAR-100

The results on CIFAR-100 highlight the consistent superiority of MrKD. For instance, on ResNet56, the baseline error of 27.56% was reduced to 26.59% by MrKD-3. Notably, on larger models like WRN-28-10, where standard LSR actually degraded performance (increasing error from 19.00% to 20.08%), MrKD3-TC achieved a significant reduction to 18.51%.

Detailed test error (%) results on CIFAR-100. MrKD variants consistently outperform the baseline and standard LSR across a wide range of architectures.

Sensitivity Analysis

Our analysis of hyperparameters revealed that the update frequency ($\kappa$) is crucial. If $\kappa$ is too small, the teacher is too similar to the student, providing little regularization. Conversely, if $\kappa$ is too large, the teacher's knowledge may be too outdated. A moderate gap provides the necessary diversity for effective self-distillation.

Discussion and Conclusion

The success of MrKD suggests that the primary benefit of Knowledge Distillation is not just mimicking a "smarter" teacher, but rather a powerful form of regularization. By forcing the model to remain consistent with its own stable historical output space, MrKD guides the optimization process toward flatter, more robust minima in the loss landscape.

In conclusion, LsrKD and MrKD offer a cost-effective way to boost model performance by leveraging internal "dark knowledge." These methods are easy to implement, parallelizable, and consistently outperform standard label smoothing across a wide range of architectures and datasets.

Summary updated

Abstract

Inspired by the strong correlation between the Label Smoothing Regularization(LSR) and Knowledge distillation(KD), we propose an algorithm LsrKD for training boost by extending the LSR method to the KD regime and applying a softer temperature. Then we improve the LsrKD by a Teacher Correction(TC) method, which manually sets a constant larger proportion for the right class in the uniform distribution teacher. To further improve the performance of LsrKD, we develop a self-distillation method named Memory-replay Knowledge Distillation (MrKD) that provides a knowledgeable teacher to replace the uniform distribution one in LsrKD. The MrKD method penalizes the KD loss between the current model's output distributions and its copies' on the training trajectory. By preventing the model learning so far from its historical output distribution space, MrKD can stabilize the learning and find a more robust minimum. Our experiments show that LsrKD can improve LSR performance consistently at no cost, especially on several deep neural networks where LSR is ineffectual. Also, MrKD can significantly improve single model training. The experiment results confirm that the TC can help LsrKD and MrKD to boost training, especially on the networks they are failed. Overall, LsrKD, MrKD, and their TC variants are comparable to or outperform the LSR method, suggesting the broad applicability of these KD methods.
BibTeX
@misc{wang2020extending,
  author = {Jiyue Wang and Pei Zhang and Wenfeng Pang and Jie Li},
  title = {{Extending Label Smoothing Regularization with Self-Knowledge Distillation}},
  year = {2020},
  eprint = {2009.05226},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.05226},
}
Figure