Abstract
Knowledge distillation (KD) is a cornerstone of model compression, yet it frequently sacrifices accuracy on rare classes to optimize for average performance. This paper introduces Robust Distillation, a framework that incorporates Group Distributionally Robust Optimization (DRO) and margin-based losses into the distillation process. We demonstrate that this approach ensures student models maintain high reliability on their most challenging classes. Notably, we find that a robust student does not strictly require a robust teacher; a well-calibrated standard teacher can effectively mentor a robust student, often resulting in a student that exceeds the teacher's own worst-class performance.
Introduction
As deep learning models are deployed in safety-critical applications like autonomous driving and medical diagnostics, average-case performance is no longer a sufficient metric. Standard KD techniques tend to exacerbate the "long-tail" problem, where performance on infrequent classes is traded off for gains on common ones. This paper addresses the challenge of preserving worst-class performance during model compression, ensuring that efficiency does not come at the cost of fairness or safety.
Methodology
The core of our approach is the integration of Group Distributionally Robust Optimization (DRO). Unlike standard distillation which minimizes average loss, Group DRO minimizes the maximum loss across all classes.
Margin-based Surrogates
To address the issue of over-parameterization—where deep models can drive training loss to zero and bypass the DRO objective—we utilize margin-based losses. These losses enforce a confidence buffer, preventing the model from becoming complacent on difficult classes during training.
Teacher-Student Dynamics
We investigate nine combinations of teacher and student objectives, categorized as Standard ($L^{std}$), Balanced ($L^{bal}$), and Robust ($L^{rob}$). A key theoretical and empirical insight is that the teacher's calibration is more critical for student robustness than the teacher's own robustness objective.
Experimental Evaluation
We evaluated the framework on several image classification benchmarks modified to simulate real-world imbalance (Long-Tailed distributions):
- CIFAR-10-LT & CIFAR-100-LT
- TinyImageNet-LT
- ImageNet-LT
These datasets were created by exponentially downsampling classes to create a significant performance gap between common and rare categories.
Key Results
Our experiments reveal that robust distillation consistently outperforms standard baselines in worst-case scenarios.
As shown in the results for CIFAR-10-LT, the robust student ($L^{rob-d}$) achieved a worst-class accuracy of 75.56%, a significant improvement over the teacher's baseline of 68.52%. This phenomenon, where the student outperforms its mentor, suggests that distillation can serve as a mechanism to rectify biases inherent in the teacher model.
Across CIFAR-100 and TinyImageNet, the robust objectives yielded the best results for the most difficult classes, often exceeding standard distillation baselines by 5-10%. Furthermore, we observed that using teacher-generated labels for validation was highly effective, reducing the need for expensive ground-truth annotations in the robustness tuning phase.
Conclusion and Future Work
Robust Distillation provides a principled way to compress models without abandoning the "tail" of the data distribution. While this work focuses on class-level robustness, future research should extend these techniques to group-level robustness involving sensitive attributes like race or gender. Additionally, exploring the optimal mixture of teacher predictions and ground-truth labels remains a promising avenue for further improving data efficiency and model reliability.
Summary updated