Teacher's pet: understanding and mitigating biases in distillation

Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv Kumar

Abstract

Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model. Several works have shown that distillation significantly boosts the student's overall performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples. We trace this behaviour to errors made by the teacher distribution being transferred to and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.

Introduction

Knowledge distillation (KD) has become a standard tool in the deep learning toolbox, primarily used for model compression by training a small "student" model to mimic a large "teacher" model. While researchers usually celebrate the boost in average accuracy, a critical question remains: does distillation help all data subgroups uniformly? In many real-world scenarios, datasets are "long-tailed," meaning a few classes have many samples while most have very few. This work identifies that in these settings, gains in average accuracy often come at the expense of minority classes or difficult subgroups.

The "Teacher's Pet" Effect: Amplification of Error

A striking finding of this research is that students do not just copy the teacher; they often amplify the teacher's mistakes. If a teacher is confidently wrong about a specific rare class, the student model, in its attempt to be a "teacher's pet," mimics those errors with even greater certainty. This leads to a degradation in fairness and robustness, particularly when the teacher makes confident mispredictions.

Comparison of margins between teacher and student models across class buckets. The student often amplifies the teacher's negative margins in difficult buckets, leading to performance degradation.

As shown in the margin analysis, the student's margin—the difference between the score of the correct class and the highest score of other classes—often becomes more negative than the teacher's in difficult class buckets. This visualizes the mechanism by which distillation can actively harm performance on challenging data points.

Proposed Methods: AdaAlpha and AdaMargin

To protect vulnerable subgroups, two primary modifications to the standard distillation objective are proposed:

  1. AdaAlpha (Adaptive Mixing Weights): Standard KD uses a global weight ($\alpha$) to balance ground-truth labels and teacher predictions. AdaAlpha introduces per-class weights ($\alpha_y$). By using a small holdout set to estimate teacher reliability, the algorithm reduces the teacher's influence on classes where it is frequently wrong but confident, allowing the student to rely more on ground-truth labels.
  2. AdaMargin (Per-class Margins): This method introduces variable margins into the loss function, increasing the penalty for misclassifying "hard" classes that the teacher struggles to model. This discourages the student from confusing rare labels with common ones.

Experimental Results

The study evaluates these phenomena across several benchmarks, including CIFAR-100-LT, ImageNet-LT, and the UCI Adult dataset. The results demonstrate that while standard distillation (Baseline) provides average gains, it frequently results in negative accuracy improvement for the worst-performing subgroups.

Accuracy improvement for average and worst-performing subgroups on CIFAR-100-LT and ImageNet. Standard distillation (Baseline) often harms the worst-performing subgroups, whereas adaptive mixing provides consistent gains.

As illustrated in the comparative bar charts, the proposed adaptive mixing techniques (AdaAlpha) successfully recover performance on these "underdog" classes. For instance, on CIFAR-100-LT, AdaAlpha boosted mean accuracy to 48.57% (compared to 45.39% for standard KD) while significantly improving performance on the worst-performing classes. Similar trends were observed on the UCI Adult dataset, where adaptive methods prevented the performance drops for specific demographic subgroups (e.g., "Black women") seen with standard distillation.

Conclusion and Limitations

Knowledge distillation is not a "free lunch." Without careful adjustment, it can introduce or exacerbate biases against rare subgroups. Adaptive methods like AdaAlpha and AdaMargin offer a path forward by making the distillation process aware of teacher reliability.

However, some hurdles remain for practical deployment. Calculating adaptive weights requires a labeled validation holdout set, which may be scarce in low-data regimes. Additionally, methods like AdaMargin can be sensitive to hyperparameter choices. Future work should explore more complex, overlapping social subgroups to ensure distillation remains a fair and robust tool for all applications.

Summary updated

Abstract

Knowledge distillation is widely used as a means of improving the performance of a relatively simple student model using the predictions from a complex teacher model. Several works have shown that distillation significantly boosts the student's overall performance; however, are these gains uniform across all data subgroups? In this paper, we show that distillation can harm performance on certain subgroups, e.g., classes with few associated samples. We trace this behaviour to errors made by the teacher distribution being transferred to and amplified by the student model. To mitigate this problem, we present techniques which soften the teacher influence for subgroups where it is less reliable. Experiments on several image classification benchmarks show that these modifications of distillation maintain boost in overall accuracy, while additionally ensuring improvement in subgroup performance.
BibTeX
@inproceedings{lukasik2022teacher,
  author = {Michal Lukasik and Srinadh Bhojanapalli and Aditya Krishna Menon and Sanjiv Kumar},
  title = {{Teacher's pet: understanding and mitigating biases in distillation}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=WDBo7y8lcJm},
}
Figure