Abstract
Knowledge distillation (KD) has gained a lot of attention in the field of model compression for edge devices thanks to its effectiveness in compressing large powerful networks into smaller lower-capacity models. Online distillation, in which both the teacher and the student are learning collaboratively, has also gained much interest due to its ability to improve on the performance of the networks involved. The Kullback-Leibler (KL) divergence ensures the proper knowledge transfer between the teacher and student. However, most online KD techniques present some bottlenecks under the network capacity gap. By cooperatively and simultaneously training the models, the KL distance becomes incapable of properly minimizing the teacher's and student's distributions. Alongside accuracy, critical edge device applications are in need of well-calibrated compact networks. Confidence calibration provides a sensible way of getting trustworthy predictions. We propose BD-KD: Balancing of Divergences for online Knowledge Distillation. We show that adaptively balancing between the reverse and forward divergences shifts the focus of the training strategy to the compact student network without limiting the teacher network's learning process. We demonstrate that, by performing this balancing design at the level of the student distillation loss, we improve upon both performance accuracy and calibration of the compact student network.
Introduction
To deploy deep learning models on edge devices like smartphones or autonomous vehicle sensors, model compression is essential. Knowledge Distillation (KD) is a primary technique where a high-capacity "teacher" model trains a compact "student" model. While traditional KD focuses on accuracy, this research addresses two critical shortcomings: the "capacity gap" between teacher and student, and model calibration.
Calibration is the alignment between a model's predicted confidence and its actual accuracy. In safety-critical applications, such as autonomous driving, an overconfident but incorrect prediction can be catastrophic. Standard KD often exacerbates overconfidence in student models. BD-KD addresses this by introducing an online distillation framework where both models learn simultaneously, utilizing a balanced divergence approach to ensure both high performance and reliable confidence estimates.
Methodology: Balancing Divergences
The core of BD-KD is the adaptive balancing of two mathematical measures of distribution distance: Forward KL Divergence and Reverse KL Divergence.
- Dual Divergences: Standard KD typically uses Forward KL, which is "mean-seeking" and tends to spread probability mass. BD-KD incorporates Reverse KL, which is "mode-seeking" and encourages the student to focus on the teacher's most likely predictions.
- Sample-wise Weighting: The framework calculates an "entropy gap" (the difference in uncertainty) for every training sample. If the student is too uncertain (high entropy), the system increases the weight of the Reverse KL. If the student is overconfident (low entropy), it increases the weight of the Forward KL to encourage exploration of alternative possibilities.
- Teacher Regularization: Unlike offline KD, the teacher in BD-KD is not fixed. It uses feedback from the student as a regularizer, which prevents the teacher from becoming overconfident and helps bridge the capacity gap.
Experimental Results
The researchers evaluated BD-KD across several standard benchmarks, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and the large-scale ImageNet dataset. The experiments utilized diverse architectures such as WideResNet (WRN), MobileNetV2, and ResNet to demonstrate the method's architecture-agnostic nature.
Accuracy Gains
BD-KD consistently outperformed state-of-the-art online distillation methods. As shown in the results table, BD-KD achieved superior Top-1 accuracy across nearly all tested configurations. For instance, on CIFAR-100 with a MobileNetV2 student, BD-KD reached 68.69% accuracy, significantly higher than Vanilla KD (60.07%) and other online methods like DML (66.23%).
Calibration Improvements
Beyond accuracy, BD-KD showed dramatic improvements in model calibration. The Expected Calibration Error (ECE)—a measure of how well a model's confidence matches its accuracy—was significantly reduced. In experiments with a ResNet20 student, the ECE dropped from 12.07% in vanilla KD to just 2.52% with BD-KD. This indicates that the resulting compact models are not only more accurate but also far more trustworthy for real-world deployment.
Conclusion
BD-KD represents a significant step forward in making compact AI both smart and reliable. By dynamically balancing forward and reverse KL divergences based on sample-wise uncertainty, the framework effectively bridges the capacity gap in online distillation. The resulting models achieve state-of-the-art accuracy while maintaining high calibration, making them ideal for safety-critical edge applications. Future work will explore extending this balanced divergence approach to more complex vision tasks like object detection and semantic segmentation.
Summary updated