Revisiting Knowledge Distillation under Distribution Shift

Songming Zhang, Ziyu Lyu, Xiaofeng Chen

Abstract

Knowledge Distillation (KD) has become a standard technique for compressing large "teacher" models into smaller, more efficient "student" models. However, most KD research assumes that training and testing data are drawn from the same distribution. This paper introduces ShiftKD, a benchmarking framework designed to evaluate KD methods under real-world distribution shifts. By analyzing over 30 state-of-the-art methods, we demonstrate that many complex KD techniques fail to maintain robustness when faced with diversity or correlation shifts. Our analysis reveals that students are often constrained by a "robustness ceiling" imposed by their teachers and that simpler distillation methods, combined with careful optimization choices like SGD, often outperform more sophisticated feature-based approaches.

Introduction: The Challenge of Distribution Shift

In the traditional machine learning paradigm, we assume the Independent and Identically Distributed (i.i.d.) condition: the data the model sees during testing will look like the data it saw during training. In practice, this assumption rarely holds. Models deployed in the real world encounter distribution shifts that can significantly degrade performance.

Two primary types of shifts are explored in this work:

  1. Diversity Shift: Changes in the visual style or domain of the data, such as moving from real-world photos to sketches or cartoons.
  2. Correlation Shift: Changes in the relationship between spurious features (like background color) and the target label. For example, if a model learns to identify digits based on their color in training, it will fail if those colors are swapped in testing.
Overview of Diversity and Correlation Shifts in the ShiftKD framework.

The central question of this research is whether the "dark knowledge" transferred from a teacher model helps a student navigate these shifts or if it merely teaches the student to replicate the teacher's own biases and vulnerabilities.

The ShiftKD Framework

To answer this question, the authors developed ShiftKD, a systematic evaluation framework that probes KD across three distinct levels:

1. Algorithmic Level

The study categorizes over 30 KD methods into three families:

  • Logit-based: These methods focus on mimicking the final output probabilities (logits) of the teacher.
  • Feature-based: These attempt to align the intermediate representations or "feature maps" of the student with those of the teacher.
  • Relation-based: These focus on preserving the structural relationships between different samples as perceived by the teacher.

2. Data Level

The framework examines how data-centric strategies affect robustness. This includes data augmentation techniques (e.g., Mixup, AutoAugment) and data pruning, where low-quality or noisy samples are removed from the distillation set to improve the signal-to-noise ratio.

3. Optimization Level

The authors investigate the impact of training hyperparameters, the use of pre-trained weights, and the choice of optimizer (specifically comparing SGD and Adam).

Experimental Setup

The benchmark utilizes five diverse datasets to simulate various shift scenarios:

  • For Diversity Shift: PACS (Photos, Art, Cartoons, Sketches), OfficeHome, and DomainNet.
  • For Correlation Shift: ColorMNIST (digits correlated with colors) and CelebA-Blond (spurious correlations between hair color and gender).

Key Insights and Findings

The Robustness Ceiling

A critical discovery is that a student's ability to handle distribution shift is fundamentally capped by the teacher's own robustness. If a teacher relies on spurious correlations (e.g., identifying a "dog" because the background is green), the student will likely inherit this flaw through the distillation process.

The Pitfalls of Low-Level Knowledge

While feature-based distillation is popular, the authors found that distilling low-level features (early layers) often harms performance under distribution shift. These layers capture domain-specific styles and textures that do not generalize well. In contrast, high-level semantic distillation—focusing on the final layers—is significantly more robust across different domains.

Optimizer Sensitivity: SGD vs. Adam

One of the most striking findings was the impact of the optimizer. Using the Adam optimizer led to a massive performance drop compared to SGD. On the PACS dataset, average accuracy plummeted from 81.1% to 61.6% when switching to Adam. This suggests that Adam tends to find "sharp minima" that are highly sensitive to even minor distribution changes, whereas SGD finds "flatter" regions that generalize better.

The Resilience of Vanilla KD

Despite the proliferation of complex state-of-the-art (SOTA) methods, "Vanilla" KD (the original logit-based method proposed by Hinton et al. in 2015) remains remarkably competitive. In many shift scenarios, complex feature-based methods like RKD or FitNet performed significantly worse than the original, simpler approach.

Key Results

On the PACS dataset, the teacher model achieved an average accuracy of 83.3%. While Vanilla KD allowed the student to reach 78.8%, more complex methods like RKD (50.4%) and FitNet (51.0%) struggled immensely. Furthermore, the study showed that data pruning—specifically using only the top 75% of "important" samples—could maintain over 95% of performance while reducing training noise, occasionally even outperforming standard data augmentation.

Conclusion and Future Directions

The ShiftKD benchmark highlights a sobering reality: more complexity in KD does not necessarily lead to more robustness. The "dark knowledge" of a teacher is a double-edged sword that can transfer both valuable logic and detrimental biases.

Future research should move toward selective distillation. Instead of blindly mimicking all aspects of a teacher, new methods should aim to filter out domain-specific "noise" (like style or spurious correlations) and focus exclusively on domain-invariant "signals" (like shape or underlying logic). The challenge for the next generation of KD is not just how much we can compress, but how we can ensure the resulting models remain reliable in a changing world.

Summary updated

Abstract

Knowledge distillation transfers knowledge from large models into small models, and has recently made remarkable achievements. However, few studies has investigated the mechanism of knowledge distillation against distribution shift. Distribution shift refers to the data distribution drifts between training and testing phases. In this paper, we reconsider the paradigm of knowledge distillation by reformulating the objective function in shift situations. Under the real scenarios, we propose a unified and systematic framework to benchmark knowledge distillation against two general distributional shifts including diversity and correlation shift. The evaluation benchmark covers more than 30 methods from algorithmic, data-driven, and optimization perspectives for five benchmark datasets. Overall, we conduct extensive experiments on the student model. We reveal intriguing observations of poor teaching performance under distribution shifts; in particular, complex algorithms and data augmentation offer limited gains in many cases.
BibTeX
@misc{zhang2023revisiting,
  author = {Songming Zhang and Ziyu Lyu and Xiaofeng Chen},
  title = {{Revisiting Knowledge Distillation under Distribution Shift}},
  year = {2023},
  eprint = {2312.16242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.16242},
}
Figure