Abstract
Knowledge Distillation (KD) has become a standard technique for compressing large "teacher" models into smaller, more efficient "student" models. However, most KD research assumes that training and testing data are drawn from the same distribution. This paper introduces ShiftKD, a benchmarking framework designed to evaluate KD methods under real-world distribution shifts. By analyzing over 30 state-of-the-art methods, we demonstrate that many complex KD techniques fail to maintain robustness when faced with diversity or correlation shifts. Our analysis reveals that students are often constrained by a "robustness ceiling" imposed by their teachers and that simpler distillation methods, combined with careful optimization choices like SGD, often outperform more sophisticated feature-based approaches.
Introduction: The Challenge of Distribution Shift
In the traditional machine learning paradigm, we assume the Independent and Identically Distributed (i.i.d.) condition: the data the model sees during testing will look like the data it saw during training. In practice, this assumption rarely holds. Models deployed in the real world encounter distribution shifts that can significantly degrade performance.
Two primary types of shifts are explored in this work:
- Diversity Shift: Changes in the visual style or domain of the data, such as moving from real-world photos to sketches or cartoons.
- Correlation Shift: Changes in the relationship between spurious features (like background color) and the target label. For example, if a model learns to identify digits based on their color in training, it will fail if those colors are swapped in testing.
The central question of this research is whether the "dark knowledge" transferred from a teacher model helps a student navigate these shifts or if it merely teaches the student to replicate the teacher's own biases and vulnerabilities.
The ShiftKD Framework
To answer this question, the authors developed ShiftKD, a systematic evaluation framework that probes KD across three distinct levels:
1. Algorithmic Level
The study categorizes over 30 KD methods into three families:
- Logit-based: These methods focus on mimicking the final output probabilities (logits) of the teacher.
- Feature-based: These attempt to align the intermediate representations or "feature maps" of the student with those of the teacher.
- Relation-based: These focus on preserving the structural relationships between different samples as perceived by the teacher.
2. Data Level
The framework examines how data-centric strategies affect robustness. This includes data augmentation techniques (e.g., Mixup, AutoAugment) and data pruning, where low-quality or noisy samples are removed from the distillation set to improve the signal-to-noise ratio.
3. Optimization Level
The authors investigate the impact of training hyperparameters, the use of pre-trained weights, and the choice of optimizer (specifically comparing SGD and Adam).
Experimental Setup
The benchmark utilizes five diverse datasets to simulate various shift scenarios:
- For Diversity Shift: PACS (Photos, Art, Cartoons, Sketches), OfficeHome, and DomainNet.
- For Correlation Shift: ColorMNIST (digits correlated with colors) and CelebA-Blond (spurious correlations between hair color and gender).
Key Insights and Findings
The Robustness Ceiling
A critical discovery is that a student's ability to handle distribution shift is fundamentally capped by the teacher's own robustness. If a teacher relies on spurious correlations (e.g., identifying a "dog" because the background is green), the student will likely inherit this flaw through the distillation process.
The Pitfalls of Low-Level Knowledge
While feature-based distillation is popular, the authors found that distilling low-level features (early layers) often harms performance under distribution shift. These layers capture domain-specific styles and textures that do not generalize well. In contrast, high-level semantic distillation—focusing on the final layers—is significantly more robust across different domains.
Optimizer Sensitivity: SGD vs. Adam
One of the most striking findings was the impact of the optimizer. Using the Adam optimizer led to a massive performance drop compared to SGD. On the PACS dataset, average accuracy plummeted from 81.1% to 61.6% when switching to Adam. This suggests that Adam tends to find "sharp minima" that are highly sensitive to even minor distribution changes, whereas SGD finds "flatter" regions that generalize better.
The Resilience of Vanilla KD
Despite the proliferation of complex state-of-the-art (SOTA) methods, "Vanilla" KD (the original logit-based method proposed by Hinton et al. in 2015) remains remarkably competitive. In many shift scenarios, complex feature-based methods like RKD or FitNet performed significantly worse than the original, simpler approach.
Key Results
On the PACS dataset, the teacher model achieved an average accuracy of 83.3%. While Vanilla KD allowed the student to reach 78.8%, more complex methods like RKD (50.4%) and FitNet (51.0%) struggled immensely. Furthermore, the study showed that data pruning—specifically using only the top 75% of "important" samples—could maintain over 95% of performance while reducing training noise, occasionally even outperforming standard data augmentation.
Conclusion and Future Directions
The ShiftKD benchmark highlights a sobering reality: more complexity in KD does not necessarily lead to more robustness. The "dark knowledge" of a teacher is a double-edged sword that can transfer both valuable logic and detrimental biases.
Future research should move toward selective distillation. Instead of blindly mimicking all aspects of a teacher, new methods should aim to filter out domain-specific "noise" (like style or spurious correlations) and focus exclusively on domain-invariant "signals" (like shape or underlying logic). The challenge for the next generation of KD is not just how much we can compress, but how we can ensure the resulting models remain reliable in a changing world.
Summary updated