Abstract
Knowledge distillation is an effective technique that transfers knowledge from a large teacher model to a shallow student. However, just like massive classification, large scale knowledge distillation also imposes heavy computational costs on training models of deep neural networks, as the softmax activations at the last layer involve computing probabilities over numerous classes. In this work, we apply the idea of importance sampling which is often used in Neural Machine Translation on large scale knowledge distillation. We present a method called dynamic importance sampling, where ranked classes are sampled from a dynamic distribution derived from the interaction between the teacher and student in full distillation. We highlight the utility of our proposal prior which helps the student capture the main information in the loss function. Our approach manages to reduce the computational cost at training time while maintaining the competitive performance on CIFAR-100 and Market-1501 person re-identification datasets.
Introduction
Knowledge distillation (KD) has become a cornerstone of model compression, enabling small "student" models to inherit the performance of large "teacher" models. However, a significant bottleneck arises in large-scale classification tasks (e.g., face recognition or natural language processing) where the number of classes $N$ is in the thousands. The final softmax layer requires $O(N)$ computations to normalize the distribution, making training both slow and memory-intensive.
While importance sampling has been used in NLP to mitigate this, traditional methods rely on static class frequencies. In KD, the "importance" of a class is dynamic, shifting as the student learns. This paper introduces Dynamic Importance Sampling (DIS), which uses an evolving probability distribution to sample only the most informative classes, accelerating training without sacrificing accuracy.
Methodology
The core of DIS is a "proposal distribution" that evolves during training. Rather than sampling classes based on static IDs, the method sorts classes by the teacher's prediction confidence (rank).
The Laplace Mixture Model
By analyzing the difference between teacher and student predictions, the authors discovered a "U-shaped" pattern of informativeness: the student learns most from classes where the teacher is highly confident (top ranks) and classes the teacher is most certain are incorrect (bottom ranks). To capture this, DIS models the sampling probability as a mixture of two Laplace distributions. Early in training, the distribution is balanced between both ends. As training progresses, the sampling focus shifts linearly toward the top-ranked classes to refine the student's accuracy.
Experimental Results
The researchers evaluated DIS on CIFAR-100 and the Market-1501 person re-identification dataset.
Performance and Efficiency
- Accuracy: On CIFAR-100, DIS using only 10 sampled classes achieved 47.30% accuracy, notably outperforming full distillation (46.35%). This suggests that the stochastic nature of DIS acts as a regularizer, helping the student generalize better than if it processed all classes.
- Speedup: On Market-1501, DIS reduced the computational time for the final softmax layer and gradients by approximately 23%.
- Stability: DIS proved significantly more stable than uniform sampling, especially when the number of sampled classes $k$ was small.
Conclusion
Dynamic Importance Sampling effectively addresses the computational bottleneck of large-scale knowledge distillation. By leveraging a rank-based Laplace Mixture Model, DIS focuses training on the most informative classes. While the overall speedup is currently bounded by the network backbone, DIS offers a robust and efficient path for scaling distillation to massive datasets with thousands or millions of classes.
Summary updated