Long Short-Term Sample Distillation

Liang Jiang, Zujie Wen, Zhongping Liang, Yafang Wang, Gerard de Melo, Zhe Li, Liangzhuang Ma, Jiaxing Zhang, Xiaolong Li, Yuan Qi

Introduction

Knowledge Distillation (KD) has long served as a powerful technique for transferring knowledge from a large, high-performance "teacher" model to a smaller, more efficient "student" model. However, traditional KD is computationally intensive, requiring the pre-training of a separate teacher. "One-generation" distillation methods attempt to mitigate this by allowing a model to learn from its own previous states. Yet, these methods face a persistent trade-off: snapshots taken too recently are too similar to provide new information, while snapshots taken too long ago provide outdated, low-quality guidance.

Long Short-Term Sample Distillation (LSTSD) resolves this paradox by employing a dual-teacher framework. By simultaneously leveraging a stable "long-term" teacher and a high-quality "short-term" teacher, LSTSD achieves state-of-the-art performance across computer vision and natural language processing tasks without the overhead of multi-stage training.

Proposed Methodology

LSTSD introduces a training regime structured around mini-generations—fixed groups of training epochs. Within this structure, every training sample is supervised by two distinct internal signals:

  1. The Long-Term Teacher ($z^L$): Derived from the model's state at the end of the previous mini-generation. It remains static throughout the current mini-generation, providing a stable reference point that maintains sufficient diversity from the current student.
  2. The Short-Term Teacher ($z^S$): Derived from the model's state in the immediately preceding epoch. It is updated every epoch, offering the most recent and high-quality cues to refine the model's learning trajectory.
Overview of the LSTSD framework, illustrating the interaction between long-term and short-term teachers across mini-generations and epochs.

A key innovation of LSTSD is its sample-level approach. Rather than using a single global snapshot for the entire dataset, the algorithm stores the probability distribution for each specific sample. Because data is shuffled every epoch, each sample effectively learns from a unique "teacher" state, maximizing the diversity of supervision signals and preventing the model from falling into an "echo chamber" of its own predictions.

Experimental Results

The researchers evaluated LSTSD on the CIFAR-100 vision dataset using ResNet and DenseNet architectures, as well as on the GLUE benchmark for NLP using BERT and CNN backbones.

Vision Performance

On CIFAR-100, LSTSD consistently outperformed vanilla training and existing one-generation methods like Mean Teacher and Snapshot Distillation. For instance, LSTSD improved ResNet-110 accuracy from a baseline of 71.47% to 73.83%.

Comparison of LSTSD against vanilla training and other distillation methods on CIFAR-100 across various network architectures.

NLP Performance

The benefits extended to language tasks. Using BERT on the MRPC (Paraphrase) task, LSTSD achieved 89.22% accuracy, a significant improvement over the 86.03% vanilla baseline. Similar gains were observed across RTE, SST-2, and CoLA datasets.

Ablation and Efficiency

Ablation studies confirmed the necessity of the dual-teacher approach. Removing the long-term teacher resulted in a 0.34% drop in ResNet-20 accuracy, while removing the short-term teacher led to a 0.60% decrease. Crucially, LSTSD adds negligible computational cost (FLOPs) during training, as it simply retrieves stored probability vectors from memory rather than performing additional forward passes through a teacher model.

Discussion and Limitations

While LSTSD offers a robust and efficient training strategy, it introduces certain considerations:

  • Hyperparameter Tuning: Performance is sensitive to the length of the mini-generation. If the interval is too short, the long-term teacher lacks diversity; if too long, its quality degrades. The authors identified 6 epochs as an optimal "sweet spot" for their experiments.
  • Memory Requirements: Storing probability distributions for every sample in a dataset can be memory-intensive. For extremely large-scale datasets with millions of samples and thousands of classes, managing these teacher vectors may require specialized memory optimization.

In conclusion, LSTSD demonstrates that dual-scale self-distillation is a highly effective way to boost model performance. By balancing stability and quality through long-term and short-term signals, it provides a practical path toward more efficient neural network training.

Summary updated

Abstract

In the past decade, there has been substantial progress at training increasingly deep neural networks. Recent advances within the teacher–student training paradigm have established that information about past training updates show promise as a source of guidance during subsequent training steps. Based on this notion, in this paper, we propose Long Short-Term Sample Distillation, a novel training policy that simultaneously leverages multiple phases of the previous training process to guide the later training updates to a neural network, while efficiently proceeding in just one single generation pass. With Long Short-Term Sample Distillation, the supervision signal for each sample is decomposed into two parts: a long-term signal and a short-term one. The long-term teacher draws on snapshots from several epochs ago in order to provide steadfast guidance and to guarantee teacher–student differences, while the short-term one yields more up-to-date cues with the goal of enabling higher-quality updates. Moreover, the teachers for each sample are unique, such that, overall, the model learns from a very diverse set of teachers. Comprehensive experimental results across a range of vision and NLP tasks demonstrate the effectiveness of this new training method.
BibTeX
@inproceedings{jiang2020long,
  author = {Liang Jiang and Zujie Wen and Zhongping Liang and Yafang Wang and Gerard de Melo and Zhe Li and Liangzhuang Ma and Jiaxing Zhang and Xiaolong Li and Yuan Qi},
  title = {{Long Short-Term Sample Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5859},
}
Figure