Abstract
Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying to design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes a "good" DA in KD? Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy. A practical metric, the stddev of teacher's mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.
Introduction
Knowledge Distillation (KD) has become a cornerstone of modern deep learning, enabling the transfer of knowledge from a large, complex "teacher" model to a smaller, more efficient "student." Traditionally, research in this area has focused on the "output side" — refining loss functions like KL-divergence or designing intermediate feature-matching losses. However, the "input side" — how we transform data before it reaches the models — remains poorly understood in the context of KD. While advanced data augmentation (DA) schemes like CutMix or Mixup are known to improve performance, the underlying reasons for their effectiveness in distillation have been unclear. This paper, presented at NeurIPS 2022, addresses this gap by providing a statistical lens on the interplay between DA and KD.
Defining a "Good" Data Augmentation for KD
The authors propose that a "good" DA scheme for knowledge distillation is one that reduces the variance of the teacher’s mean output probability. They derive a practical, student-invariant metric called T. stddev (the standard deviation of the teacher’s mean probability).
To calculate T. stddev, one takes a batch of augmented images, passes them through the fixed teacher, averages the output probabilities, and calculates the standard deviation across classes. A lower T. stddev indicates a "stronger" and more effective DA scheme for KD. Crucially, this metric allows for evaluating DA effectiveness without the expensive process of training a student model.
CutMixPick: Entropy-Based Sample Selection
Building on their theoretical framework, the authors introduce CutMixPick, an enhancement to the popular CutMix algorithm. Instead of using all generated CutMix samples, CutMixPick uses Shannon entropy as a filter. By generating a pool of augmented samples and "picking" those where the teacher is most uncertain (highest entropy), the student is provided with the most informative "knowledge" to absorb. This approach treats DA in KD as a "function matching" task, where the teacher defines the label for the augmented view, allowing for more aggressive transformations than standard training.
Key Insights and Empirical Findings
The research reveals several critical insights into the nature of KD and DA:
- Student-Invariance: The effectiveness of a DA scheme is largely independent of the student model. One can predict how well a DA will work for any student just by observing the teacher's reaction.
- Diversity over Accuracy: A good DA in KD doesn't necessarily need to preserve the original ground-truth label. Instead, it should produce diverse teacher outputs (low covariance). This explains why CutMix often outperforms Mixup in distillation settings.
- The Need for Prolonged Training: Stronger DA provides more information, requiring the student more time to "digest" it. As shown in the figure below, students trained with KD and strong DA continue to improve long after standard training (CE) would have begun to overfit.
- Label Disagreement: On ImageNet, for over 50% of CutMix samples, the teacher's predicted label disagreed with the mixed ground-truth label. In KD, the teacher provides more accurate supervision by correcting these labels, whereas in standard training, they would act as noise.
Results and Conclusion
The proposed T. stddev metric shows a strong positive correlation (p-value < 5%) with student test loss across various teacher-student pairs. On benchmarks like CIFAR100, combining CutMixPick with prolonged training (960 epochs) yielded significant gains. For instance, a ResNet8x4 student trained with a ResNet32x4 teacher improved from 73.33% with standard KD to 75.12% with CutMixPick.
While the correlation was slightly weaker on full ImageNet — possibly due to the inherent difficulty of KD on ImageNet — the findings demonstrate that substantial performance gains can be harvested simply by selecting better DA schemes. This work shifts the focus of KD research from complex loss functions to the often-overlooked power of input-side data transformations.
Summary updated