Abstract
Inspired by the strong correlation between Label Smoothing Regularization (LSR) and Knowledge Distillation (KD), we propose LsrKD, an algorithm that extends LSR into the KD regime by applying a softer temperature. We further enhance this with Teacher Correction (TC), which manually adjusts the teacher distribution to favor the correct class. To push performance further, we introduce Memory-replay Knowledge Distillation (MrKD), a self-distillation method that uses historical model snapshots as teachers. MrKD penalizes the divergence between current outputs and previous iterations, stabilizing training and finding more robust minima. Our experiments demonstrate that LsrKD and MrKD consistently outperform standard LSR across various deep neural networks, even in cases where LSR fails to provide benefits.
Introduction
In the quest to train more robust deep learning models, researchers often turn to two popular techniques: Label Smoothing Regularization (LSR) and Knowledge Distillation (KD). LSR prevents overconfidence by softening "one-hot" labels with a uniform distribution, while KD transfers "dark knowledge" from a complex teacher model to a simpler student. While they appear distinct, they are two sides of the same coin: both provide a more informative target distribution than hard labels.
This paper bridges these concepts by reformulating LSR as a special case of KD. We introduce LsrKD and its evolution, Memory-replay Knowledge Distillation (MrKD). These methods leverage a model's own training trajectory to provide a knowledgeable teacher without the computational overhead of pre-training an external model.
Methodology
1. LsrKD: Softening the Uniform Teacher
Standard LSR can be viewed as KD where the teacher is a fixed uniform distribution. However, LSR typically uses a "hard" softmax. We propose LsrKD, which introduces a temperature parameter ($\tau$) to the softmax function. By increasing $\tau$, the uniform distribution becomes "softer," revealing more structural information that acts as a more effective regularizer than standard LSR.
2. Teacher Correction (TC)
A purely uniform teacher can be "lifeless," treating all classes equally regardless of the true label. Teacher Correction (TC) addresses this by manually setting a slightly higher constant probability for the correct class within the teacher distribution. This provides a more reliable signal while maintaining the benefits of smoothing.
3. Memory-replay Knowledge Distillation (MrKD)
The most significant contribution is MrKD. Instead of a static uniform distribution, MrKD uses historical backups of the model itself as the teacher. By saving model weights every $\kappa$ epochs, we create a "memory" of the model's past. The current model is then trained to match the output distributions of these historical versions.
As shown in Figure 1, MrKD (d, e) provides a much richer teacher distribution compared to the uniform approach of LSR (a). This "dark knowledge" captures the relationships between classes as learned by the model earlier in its training trajectory.
Experimental Results
We evaluated our methods on CIFAR-10, CIFAR-100, and CINIC-10 using various architectures, including ResNet, WideResNet (WRN), and ResNeXt.
Performance on CIFAR-100
The results on CIFAR-100 highlight the consistent superiority of MrKD. For instance, on ResNet56, the baseline error of 27.56% was reduced to 26.59% by MrKD-3. Notably, on larger models like WRN-28-10, where standard LSR actually degraded performance (increasing error from 19.00% to 20.08%), MrKD3-TC achieved a significant reduction to 18.51%.
Sensitivity Analysis
Our analysis of hyperparameters revealed that the update frequency ($\kappa$) is crucial. If $\kappa$ is too small, the teacher is too similar to the student, providing little regularization. Conversely, if $\kappa$ is too large, the teacher's knowledge may be too outdated. A moderate gap provides the necessary diversity for effective self-distillation.
Discussion and Conclusion
The success of MrKD suggests that the primary benefit of Knowledge Distillation is not just mimicking a "smarter" teacher, but rather a powerful form of regularization. By forcing the model to remain consistent with its own stable historical output space, MrKD guides the optimization process toward flatter, more robust minima in the loss landscape.
In conclusion, LsrKD and MrKD offer a cost-effective way to boost model performance by leveraging internal "dark knowledge." These methods are easy to implement, parallelizable, and consistently outperform standard label smoothing across a wide range of architectures and datasets.
Summary updated