Abstract
Knowledge distillation (KD) is a widely adopted technique for compressing large-scale neural networks into lightweight student models. However, standard KD often causes the student to inherit or even amplify the teacher's biases, leading to poor performance on underrepresented subgroups. This paper introduces GroupDistil, a novel framework that integrates KD with distributionally robust optimization (DRO). By dynamically prioritizing minority groups during the distillation process, GroupDistil produces student models that are significantly more robust to sub-population shifts in both natural and medical imaging domains.
Introduction
In real-world applications, datasets are frequently imbalanced. A model trained on medical imaging, for instance, may encounter thousands of samples from a dominant scanner manufacturer but only a few from a minority vendor. This phenomenon, known as sub-population shift, often leads standard deep learning models to optimize for the majority at the expense of minority groups.
While Knowledge Distillation is the standard for model compression, recent studies indicate that student models often adopt the "bad habits" of their teachers, including poor generalization on rare subgroups. GroupDistil addresses this by making the distillation process "group-aware," ensuring that the student maintains high accuracy even on the most challenging or underrepresented slices of the data.
Methodology
GroupDistil modifies the standard distillation objective by incorporating principles from Group Distributionally Robust Optimization (groupDRO). The method focuses on minimizing the loss of the worst-performing group rather than the average loss across the entire dataset.
Per-Group Weighting
Instead of treating every training sample equally, GroupDistil assigns a dynamic weight ($w_d$) to each domain or group. This allows the model to shift its focus based on real-time performance metrics during training.
Dynamic Optimization
The training process follows a two-step iterative cycle:
- Loss Calculation: The distillation loss is calculated for each group, incorporating both standard cross-entropy and the Kullback–Leibler (KL) divergence from the teacher's soft targets.
- Weight Update: Group weights are updated using exponentiated gradient ascent. If the student performs poorly on a specific group, that group's weight is increased, forcing the model to prioritize those samples in the subsequent iteration.
This approach ensures that the student model does not merely mimic the teacher's average behavior but specifically catches up on groups where the teacher or a standard student would typically struggle.
Experimental Evaluation
The researchers evaluated GroupDistil on two distinct benchmarks representing different types of sub-population shifts:
- Waterbirds: A benchmark for spurious correlations where the task is to classify birds (landbird vs. waterbird). The background (land vs. water) serves as a confounding factor, as most waterbirds are pictured on water. Robustness here requires the model to ignore the background and focus on the bird itself.
- M&Ms (Multi-Centre, Multi-Vendor, Multi-Disease): A medical dataset containing cardiac MRIs from four different scanner vendors (Siemens, Philips, GE, and Canon). The authors utilized this for a binary classification task (Hypertrophic Cardiomyopathy vs. Healthy) to test robustness across different hardware domains.
Key Insights and Results
Performance Gains
GroupDistil demonstrated superior performance in worst-group accuracy across both datasets:
- Waterbirds: GroupDistil achieved a worst-group accuracy of 81.7%, surpassing vanilla KD (78.0%) and even outperforming a ResNet-18 trained from scratch using groupDRO (79.6%).
- Medical Imaging: In the most challenging split of the M&Ms dataset (testing on the underrepresented Domain C), GroupDistil reached 62.8% accuracy, compared to 55.8% for standard KD and 53.3% for a model trained from scratch.
Transferability of Robustness
A significant finding is that while a robust teacher (pre-trained with groupDRO) provides a better signal, the student still requires a specialized loss function like GroupDistil to effectively capture and retain that robustness. Interestingly, the distilled student often outperformed the "from-scratch" baseline, suggesting that the "dark knowledge" from the teacher provides a helpful inductive bias that standard training lacks.
Stability
In the medical M&Ms dataset, standard KD exhibited high variance in results due to limited data in certain splits. GroupDistil remained significantly more stable and robust across different random seeds, making it a more reliable choice for sensitive applications like healthcare.
Conclusion and Future Work
GroupDistil provides a robust framework for model compression that does not sacrifice the performance of minority groups. By combining the efficiency of knowledge distillation with the fairness of distributionally robust optimization, it creates models that are both lightweight and equitable.
A current limitation of the approach is the requirement for fully labeled group data; the model must know which group every training sample belongs to. Future research will explore latent domain inference, aiming to develop methods that can automatically identify these subgroups without explicit labels, thereby extending the reach of robust distillation to uncurated, real-world datasets.
Summary updated