Introduction
Knowledge Distillation (KD) has traditionally relied on a large, pre-trained "teacher" model to guide the learning of a smaller "student." However, obtaining a high-quality pre-trained teacher is not always feasible. Online Knowledge Distillation (OKD) addresses this by training a group of student networks simultaneously from scratch.
A significant challenge in OKD is the "blind leading the blind" phenomenon: in early training stages, student predictions are noisy and inconsistent, leading to unstable training. This paper introduces Temporal-Spatial Boosting (TSB), a framework designed to stabilize this process by extracting a superior teacher signal from both the historical evolution and the collective intelligence of the student group.
The Temporal-Spatial Boosting Framework
TSB utilizes two primary modules to refine the distillation targets during training:
- Temporal Accumulator (TA): This module maintains a running average of predictions for each training sample across epochs using a momentum-based update. By capturing the long-term learning trend, the TA filters out stochastic noise, providing a more stable historical target for the network.
- Spatial Integrator (SI): The SI acts as an on-the-fly ensemble, aggregating predictions from all peer networks (typically via simple averaging). This ensemble provides a robust "wisdom of the crowd" target that is consistently more accurate than any individual network.
Each network in the ensemble is optimized using a composite loss function that combines standard cross-entropy with a temporal loss ($L_{ta}$) and a spatial loss ($L_{si}$), ensuring that each model learns from ground truth, its own history, and its peers simultaneously.
Experimental Results
The researchers evaluated TSB on standard benchmarks including CIFAR-10, CIFAR-100, and ImageNet across various architectures like ResNet, Wide ResNet (WRN), and MobileNetV2.
- CIFAR-100: TSB enabled a ResNet-32 to achieve 74.01% accuracy, a significant jump from the 71.20% achieved by standard Deep Mutual Learning (DML) and 68.99% for independent training.
- ImageNet: On a large scale, TSB improved ResNet-18 performance by 1.5% over the baseline, outperforming other state-of-the-art online KD methods like KDCL and ONE.
- Architecture Agnostic: The method proved effective even when distilling knowledge between different architectures (e.g., MobileNet to Wide ResNet).
Key Insights
- History as a Stabilizer: As shown in the TA visualization, smoothing predictions over time prevents students from chasing temporary noise, leading to a smoother optimization landscape and fewer accuracy drops.
- Continuous Ensemble Benefit: The Spatial Integrator demonstrates that the ensemble average consistently outperforms individual networks from the very beginning of training, providing a high-quality teacher signal on the fly.
Conclusion
Temporal-Spatial Boosting offers a robust solution to the instabilities of online knowledge distillation. By leveraging both temporal smoothing and spatial integration, TSB creates a high-quality teacher signal without the need for pre-training. While the method introduces some memory overhead for storing accumulated predictions, its significant accuracy gains and stability make it a powerful tool for training efficient neural networks.
Summary updated