ShiftKD: Benchmarking knowledge distillation under distribution shift

Songming Zhang, Yuxiao Luo, Ziyu Lyu, Xiaofeng Chen

TL;DR

This paper introduces SHIFTKD, the first comprehensive benchmark for evaluating Knowledge Distillation (KD) methods under distribution shift. They found that while KD can help, many methods struggle with shifts, and vanilla KD often performs surprisingly well, highlighting the need for more robust KD algorithms.

What Problem Does This Paper Address?

Machine learning models, especially large ones, are computationally expensive. Knowledge Distillation (KD) aims to transfer knowledge from a large "teacher" model to a smaller "student" model, making it more efficient for deployment. However, real-world applications often encounter "distribution shifts," where the data distribution during inference differs from the training data. This shift can significantly degrade model performance, and it's unclear how well existing KD methods handle this challenge. Current KD research largely assumes an "i.i.d." (independent and identically distributed) setting, which doesn't reflect real-world scenarios. This paper addresses this gap by systematically evaluating KD methods under various distribution shifts.

Methods

The core contribution is the SHIFTKD framework, a benchmark designed to evaluate KD under distribution shift. This framework involves:

  • Two types of distribution shifts: Diversity shift (changes in data characteristics like style) and Correlation shift (changes in the relationship between features and labels, e.g., color-label correlation).
  • A systematic evaluation of over 30 SOTA KD methods: These methods are categorized into three types based on the knowledge they transfer:
  • Logit-based: Mimicking the teacher's final layer outputs.
  • Feature-based: Matching intermediate layer representations.
  • Relation-based: Transferring relationships between layers or samples.
  • Analysis of data manipulation techniques: Investigating how data augmentation and pruning affect KD performance under shifts.
  • Examination of optimization options: Studying the impact of hyperparameters, pre-training, and optimizers (like Adam vs. SGD).
  • Fine-grained analysis: Investigating factors like teacher model capacity, student architecture, and the role of model calibration.

Datasets

The authors use five popular datasets to simulate distribution shifts:

  • PACS: A dataset with diverse image domains.
  • OfficeHome: Another dataset with domain shifts.
  • DomainNet: A large-scale dataset with many domains.
  • ColorMNIST: A modified MNIST dataset where digit class is correlated with color, creating a correlation shift.
  • CelebA-Blond: A subset of CelebA where blonde hair is correlated with gender in training but not in testing, also creating a correlation shift.

Key Insights and Ideas

  • Distribution shift is a major challenge for KD: Many KD methods, even complex ones, struggle to maintain performance when the data distribution shifts, often performing no better than vanilla KD.
  • Vanilla KD can be surprisingly effective: In many scenarios, the basic KD method (Hinton et al., 2015) performs comparably to or even better than more sophisticated techniques, especially under correlation shifts.
  • Low-level knowledge can be misleading: Forcing students to mimic intermediate teacher layers can be detrimental under distribution shifts, as these layers might capture spurious correlations. High-level semantic features are more robust.
  • Teacher robustness is a bottleneck: A student model's robustness ceiling is largely constrained by the teacher model's robustness. If the teacher has biases, KD tends to perpetuate them.
  • Data augmentation is not a silver bullet: While helpful, data augmentation's effectiveness is context-dependent and can be less impactful when combined with KD compared to its standalone use. Random-based augmentation seems more reliable than generative methods in this context.
  • Data pruning can be beneficial: Selecting high-quality data samples through pruning can sometimes outperform data augmentation, especially in correlation-shifted datasets.
  • SGD might be better than Adam for KD under shifts: The authors observed that SGD's ability to find flatter minima makes it more robust to distribution shifts compared to Adam, which can converge to sharp minima that amplify sensitivity.
  • Calibration matters: Model calibration (how well predicted confidence matches actual accuracy) is crucial for assessing robustness, as high accuracy alone doesn't guarantee good performance under shifts.

Key Results

  • No single KD method consistently outperforms others across all shifts: Performance varies significantly depending on the dataset and the type of shift.
  • Improvements over vanilla KD are often marginal: Many advanced KD methods show only small gains over vanilla KD, and some even underperform.
  • Data augmentation can improve worst-group accuracy: Techniques like AutoAugment and RandAugment show improvements, but their impact is not uniform across all datasets and shifts.
  • Data pruning can be effective: Random pruning achieved over 95% of pre-pruning performance and sometimes outperformed augmentation on correlation shifts.
  • Teacher model capacity matters, but not always: While larger teachers can help, their biases can still be transferred to students.
  • Student architecture influences KD effectiveness: Different student architectures (e.g., MobileNet vs. ResNet) have varying sensitivities to KD.

Limitations and Future Work

  • Context-specific findings: The authors emphasize that their negative claims (e.g., "Method A doesn't work") are setting-specific and context-constrained.
  • Need for more robust KD algorithms: The current KD methods are largely vulnerable to spurious correlations and distribution shifts.
  • Exploring novel data-level methods: Enhancing student model robustness through new data-level techniques is a promising direction.
  • Further study on training strategies: Pre-training and optimizer choices are important for KD under distribution shifts and warrant more research.
  • Real-world deployment challenges: Pinpointing distribution shifts and deploying lightweight models on end-sides efficiently remains an important future area.
  • Information theory insights: The paper connects KD to information theory, suggesting that filtering context-variant components is key for student robustness. Future work could further explore this connection.

Summary updated

BibTeX
@article{zhang2025shiftkd,
  author = {Songming Zhang and Yuxiao Luo and Ziyu Lyu and Xiaofeng Chen},
  title = {{ShiftKD: Benchmarking knowledge distillation under distribution shift}},
  journal = {Neural Networks},
  year = {2025},
  doi = {10.1016/j.neunet.2025.107838},
}
Figure