Online Knowledge Distillation by Temporal-Spatial Boosting

Chengcheng Li, Zi Wang, Hairong Qi

Introduction

Knowledge Distillation (KD) has traditionally relied on a large, pre-trained "teacher" model to guide the learning of a smaller "student." However, obtaining a high-quality pre-trained teacher is not always feasible. Online Knowledge Distillation (OKD) addresses this by training a group of student networks simultaneously from scratch.

A significant challenge in OKD is the "blind leading the blind" phenomenon: in early training stages, student predictions are noisy and inconsistent, leading to unstable training. This paper introduces Temporal-Spatial Boosting (TSB), a framework designed to stabilize this process by extracting a superior teacher signal from both the historical evolution and the collective intelligence of the student group.

The Temporal-Spatial Boosting Framework

TSB utilizes two primary modules to refine the distillation targets during training:

  1. Temporal Accumulator (TA): This module maintains a running average of predictions for each training sample across epochs using a momentum-based update. By capturing the long-term learning trend, the TA filters out stochastic noise, providing a more stable historical target for the network.
  2. Spatial Integrator (SI): The SI acts as an on-the-fly ensemble, aggregating predictions from all peer networks (typically via simple averaging). This ensemble provides a robust "wisdom of the crowd" target that is consistently more accurate than any individual network.
The TSB architecture showing how multiple models interact with Temporal Accumulators and a Spatial Integrator to generate refined loss signals.

Each network in the ensemble is optimized using a composite loss function that combines standard cross-entropy with a temporal loss ($L_{ta}$) and a spatial loss ($L_{si}$), ensuring that each model learns from ground truth, its own history, and its peers simultaneously.

Experimental Results

The researchers evaluated TSB on standard benchmarks including CIFAR-10, CIFAR-100, and ImageNet across various architectures like ResNet, Wide ResNet (WRN), and MobileNetV2.

  • CIFAR-100: TSB enabled a ResNet-32 to achieve 74.01% accuracy, a significant jump from the 71.20% achieved by standard Deep Mutual Learning (DML) and 68.99% for independent training.
  • ImageNet: On a large scale, TSB improved ResNet-18 performance by 1.5% over the baseline, outperforming other state-of-the-art online KD methods like KDCL and ONE.
  • Architecture Agnostic: The method proved effective even when distilling knowledge between different architectures (e.g., MobileNet to Wide ResNet).

Key Insights

Visualizing the smoothing effect of the Temporal Accumulator on logits (left) and the accuracy advantage of the Spatial Integrator throughout training (right).
  • History as a Stabilizer: As shown in the TA visualization, smoothing predictions over time prevents students from chasing temporary noise, leading to a smoother optimization landscape and fewer accuracy drops.
  • Continuous Ensemble Benefit: The Spatial Integrator demonstrates that the ensemble average consistently outperforms individual networks from the very beginning of training, providing a high-quality teacher signal on the fly.

Conclusion

Temporal-Spatial Boosting offers a robust solution to the instabilities of online knowledge distillation. By leveraging both temporal smoothing and spatial integration, TSB creates a high-quality teacher signal without the need for pre-training. While the method introduces some memory overhead for storing accumulated predictions, its significant accuracy gains and stability make it a powerful tool for training efficient neural networks.

Summary updated

Abstract

Online knowledge distillation (KD) mutually trains a group of student networks from scratch in a peer-teaching manner, eliminating the need for pre-trained teacher models. However, supervision from peers can be noisy, especially in the early stage of training. In this paper, we propose a novel method for online knowledge distillation by temporal-spatial boosting (TSB). The proposed method constructs superior “teachers” with two modules, temporal accumulator and spatial integrator. Specifically, the temporal accumulator leverages the previous outputs of networks during training and produces a representative prediction over all classes. Instead of merely imitating the outputs of other networks as in vanilla online KD, we further propose the so-called spatial integrator that consolidates the knowledge learned by all networks and yields a stronger instructor. The operations of these two modules are simple and straightforward, which can be computed efficiently on the fly during training. The proposed method can improve the efficiency of transferring effective knowledge as well as stabilize the training process. Experimental results on various benchmark datasets and network structures validate the effectiveness of the proposed method over the state-of-the-art.
BibTeX
@inproceedings{li2022online,
  author = {Chengcheng Li and Zi Wang and Hairong Qi},
  title = {{Online Knowledge Distillation by Temporal-Spatial Boosting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00354},
}
Figure