Robust Distillation for Worst-class Performance

Serena Wang, Harikrishna Narasimhan, Yichen Zhou, Sara Hooker, Michal Lukasik, Aditya Krishna Menon

Abstract

Knowledge distillation (KD) is a cornerstone of model compression, yet it frequently sacrifices accuracy on rare classes to optimize for average performance. This paper introduces Robust Distillation, a framework that incorporates Group Distributionally Robust Optimization (DRO) and margin-based losses into the distillation process. We demonstrate that this approach ensures student models maintain high reliability on their most challenging classes. Notably, we find that a robust student does not strictly require a robust teacher; a well-calibrated standard teacher can effectively mentor a robust student, often resulting in a student that exceeds the teacher's own worst-class performance.

Introduction

As deep learning models are deployed in safety-critical applications like autonomous driving and medical diagnostics, average-case performance is no longer a sufficient metric. Standard KD techniques tend to exacerbate the "long-tail" problem, where performance on infrequent classes is traded off for gains on common ones. This paper addresses the challenge of preserving worst-class performance during model compression, ensuring that efficiency does not come at the cost of fairness or safety.

Methodology

The core of our approach is the integration of Group Distributionally Robust Optimization (DRO). Unlike standard distillation which minimizes average loss, Group DRO minimizes the maximum loss across all classes.

Margin-based Surrogates

To address the issue of over-parameterization—where deep models can drive training loss to zero and bypass the DRO objective—we utilize margin-based losses. These losses enforce a confidence buffer, preventing the model from becoming complacent on difficult classes during training.

Teacher-Student Dynamics

We investigate nine combinations of teacher and student objectives, categorized as Standard ($L^{std}$), Balanced ($L^{bal}$), and Robust ($L^{rob}$). A key theoretical and empirical insight is that the teacher's calibration is more critical for student robustness than the teacher's own robustness objective.

Experimental Evaluation

We evaluated the framework on several image classification benchmarks modified to simulate real-world imbalance (Long-Tailed distributions):

  • CIFAR-10-LT & CIFAR-100-LT
  • TinyImageNet-LT
  • ImageNet-LT

These datasets were created by exponentially downsampling classes to create a significant performance gap between common and rare categories.

Key Results

Our experiments reveal that robust distillation consistently outperforms standard baselines in worst-case scenarios.

Table 1: Worst-class accuracy results for CIFAR-10-LT and CIFAR-100-LT across various teacher-student objective combinations.

As shown in the results for CIFAR-10-LT, the robust student ($L^{rob-d}$) achieved a worst-class accuracy of 75.56%, a significant improvement over the teacher's baseline of 68.52%. This phenomenon, where the student outperforms its mentor, suggests that distillation can serve as a mechanism to rectify biases inherent in the teacher model.

Table 2: Performance metrics for standard CIFAR-10, CIFAR-100, and TinyImageNet, demonstrating the consistency of robust objectives.

Across CIFAR-100 and TinyImageNet, the robust objectives yielded the best results for the most difficult classes, often exceeding standard distillation baselines by 5-10%. Furthermore, we observed that using teacher-generated labels for validation was highly effective, reducing the need for expensive ground-truth annotations in the robustness tuning phase.

Conclusion and Future Work

Robust Distillation provides a principled way to compress models without abandoning the "tail" of the data distribution. While this work focuses on class-level robustness, future research should extend these techniques to group-level robustness involving sensitive attributes like race or gender. Additionally, exploring the optimal mixture of teacher predictions and ground-truth labels remains a promising avenue for further improving data efficiency and model reliability.

Summary updated

Abstract

Knowledge distillation has proven to be an effective technique in improving the performance a student model using predictions from a teacher model. However, recent work has shown that gains in average efficiency are not uniform across subgroups in the data, and in particular can often come at the cost of accuracy on rare subgroups and classes. To preserve strong performance across classes that may follow a long-tailed distribution, we develop distillation techniques that are tailored to improve the student's worst-class performance. Specifically, we introduce robust optimization objectives in different combinations for the teacher and student, and further allow for training with any tradeoff between the overall accuracy and the robust worst-class objective. We show empirically that our robust distillation techniques not only achieve better worst-class performance, but also lead to Pareto improvement in the tradeoff between overall performance and worst-class performance compared to other baseline methods. Theoretically, we provide insights into what makes a good teacher when the goal is to train a robust student.
BibTeX
@misc{wang2022robust,
  author = {Serena Wang and Harikrishna Narasimhan and Yichen Zhou and Sara Hooker and Michal Lukasik and Aditya Krishna Menon},
  title = {{Robust Distillation for Worst-class Performance}},
  year = {2022},
  eprint = {2206.06479},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.06479},
}
Figure