Introduction
Knowledge Distillation (KD) has long served as a powerful technique for transferring knowledge from a large, high-performance "teacher" model to a smaller, more efficient "student" model. However, traditional KD is computationally intensive, requiring the pre-training of a separate teacher. "One-generation" distillation methods attempt to mitigate this by allowing a model to learn from its own previous states. Yet, these methods face a persistent trade-off: snapshots taken too recently are too similar to provide new information, while snapshots taken too long ago provide outdated, low-quality guidance.
Long Short-Term Sample Distillation (LSTSD) resolves this paradox by employing a dual-teacher framework. By simultaneously leveraging a stable "long-term" teacher and a high-quality "short-term" teacher, LSTSD achieves state-of-the-art performance across computer vision and natural language processing tasks without the overhead of multi-stage training.
Proposed Methodology
LSTSD introduces a training regime structured around mini-generations—fixed groups of training epochs. Within this structure, every training sample is supervised by two distinct internal signals:
- The Long-Term Teacher ($z^L$): Derived from the model's state at the end of the previous mini-generation. It remains static throughout the current mini-generation, providing a stable reference point that maintains sufficient diversity from the current student.
- The Short-Term Teacher ($z^S$): Derived from the model's state in the immediately preceding epoch. It is updated every epoch, offering the most recent and high-quality cues to refine the model's learning trajectory.
A key innovation of LSTSD is its sample-level approach. Rather than using a single global snapshot for the entire dataset, the algorithm stores the probability distribution for each specific sample. Because data is shuffled every epoch, each sample effectively learns from a unique "teacher" state, maximizing the diversity of supervision signals and preventing the model from falling into an "echo chamber" of its own predictions.
Experimental Results
The researchers evaluated LSTSD on the CIFAR-100 vision dataset using ResNet and DenseNet architectures, as well as on the GLUE benchmark for NLP using BERT and CNN backbones.
Vision Performance
On CIFAR-100, LSTSD consistently outperformed vanilla training and existing one-generation methods like Mean Teacher and Snapshot Distillation. For instance, LSTSD improved ResNet-110 accuracy from a baseline of 71.47% to 73.83%.
NLP Performance
The benefits extended to language tasks. Using BERT on the MRPC (Paraphrase) task, LSTSD achieved 89.22% accuracy, a significant improvement over the 86.03% vanilla baseline. Similar gains were observed across RTE, SST-2, and CoLA datasets.
Ablation and Efficiency
Ablation studies confirmed the necessity of the dual-teacher approach. Removing the long-term teacher resulted in a 0.34% drop in ResNet-20 accuracy, while removing the short-term teacher led to a 0.60% decrease. Crucially, LSTSD adds negligible computational cost (FLOPs) during training, as it simply retrieves stored probability vectors from memory rather than performing additional forward passes through a teacher model.
Discussion and Limitations
While LSTSD offers a robust and efficient training strategy, it introduces certain considerations:
- Hyperparameter Tuning: Performance is sensitive to the length of the mini-generation. If the interval is too short, the long-term teacher lacks diversity; if too long, its quality degrades. The authors identified 6 epochs as an optimal "sweet spot" for their experiments.
- Memory Requirements: Storing probability distributions for every sample in a dataset can be memory-intensive. For extremely large-scale datasets with millions of samples and thousands of classes, managing these teacher vectors may require specialized memory optimization.
In conclusion, LSTSD demonstrates that dual-scale self-distillation is a highly effective way to boost model performance. By balancing stability and quality through long-term and short-term signals, it provides a practical path toward more efficient neural network training.
Summary updated