Knowledge Distillation as Semiparametric Inference

Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, Lester Mackey

Knowledge Distillation (KD) is a staple technique in model compression, where a small "student" model is trained to mimic the outputs of a larger, more accurate "teacher" model. Curiously, students often end up more accurate than if they were trained directly on the original labels. While the empirical success of KD is well-documented, the field has lacked a rigorous mathematical framework to explain exactly why it succeeds or when it fails.

This paper provides a fresh theoretical lens on KD by framing it as a semiparametric inference problem. By treating the teacher’s predictions as a "plug-in" estimate for unknown true probabilities, the authors identify two major failure modes—teacher overfitting and underfitting—and introduce two powerful, statistically grounded fixes: cross-fitting and loss correction.

The Problem: Data Reuse and Error Propagation

The authors identify a critical gap in our understanding of how teacher errors propagate to the student. Whether a teacher memorizes noise (overfitting) or is too simple to capture the data's complexity (underfitting), these errors inevitably impact the student.

Specifically, they tackle the "data reuse" problem. Because the teacher and student are typically trained on the exact same dataset, the student is prone to inheriting the teacher's overfitting tendencies rather than learning the underlying, generalizable patterns. If a teacher overfits, its "soft labels" on the training set become too close to the hard labels. The student then effectively performs standard supervised learning, losing the benefit of the "dark knowledge" (the relative probabilities between classes) that KD is supposed to provide.

A Semiparametric Framework

The core innovation of this work is casting KD as a semiparametric inference problem. In this statistical framework:

  • The Target is the optimal student model.
  • The Nuisance is the unknown true "Bayes" class probabilities (the ground truth distribution).
  • The Plug-in is the teacher’s predicted probabilities.

Using this perspective, the authors derive new error guarantees and propose two enhancements to standard KD:

  1. Cross-fitting (Combating Overfitting): To prevent the student from learning the teacher's memorized noise, the dataset is split into folds. The teacher is trained on one fold, and its predictions are used to train the student on a different fold.
  2. Loss Correction (Combating Underfitting): The authors introduce a "$\gamma$-corrected" loss inspired by orthogonal machine learning. They add a correction term to standard losses (like Squared Error Logit or Annealed Cross-Entropy) that uses the actual hard labels ($y$) to "nudge" the teacher's soft labels closer to the truth. This balances the teacher's bias against the student's variance.

Key Results and Performance

The authors validated their "Enhanced KD" approach across diverse domains, including five real-world tabular datasets (such as FICO, Adult, and Higgs) and image data (CIFAR-10 using ResNet architectures).

Test AUC vs. Student's number of trees, showing Enhanced KD and Cross-fit KD achieving higher performance and faster convergence than standard KD.
  • Tabular Improvements: On the tabular datasets, combining cross-fitting and loss correction improved student performance by up to 4% AUC over vanilla KD.
  • Image Robustness: On CIFAR-10, when the teacher was forced to overfit by turning off data augmentation, Enhanced KD improved accuracy by up to 1.5% and significantly lowered the test loss compared to standard distillation.
  • Overcoming Teacher Underfitting: Loss correction proved exceptionally powerful when the teacher model was weak. In experiments on the Adult dataset, loss correction allowed the student to actually outperform its own teacher when the teacher was intentionally restricted to shallow decision trees.
Test AUC vs. Teacher's max tree depth, demonstrating that Enhanced KD allows the student to outperform a restricted, underfitting teacher.

Limitations and Future Directions

While Enhanced KD offers strong theoretical guarantees and empirical gains, it comes with practical tradeoffs. The loss correction mechanism introduces a hyperparameter ($\alpha$) to balance bias and variance, requiring additional tuning. Furthermore, cross-fitting requires training multiple versions of the teacher on different data folds, which increases the initial training computational overhead (though it does not affect the student's final inference speed).

Looking forward, the authors suggest exploring other semiparametric tools, such as Targeted Maximum Likelihood Estimation (TMLE), and investigating whether this inference framework can mathematically explain the curious success of "self-distillation," where a model acts as its own teacher.

Summary updated

Abstract

A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements—cross-fitting and loss correction—to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.
BibTeX
@inproceedings{dao2021knowledge,
  author = {Tri Dao and Govinda M Kamath and Vasilis Syrgkanis and Lester Mackey},
  title = {{Knowledge Distillation as Semiparametric Inference}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=m4UCf24r0Y},
}
Figure