Abstract
A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation (KD) process often leads to higher accuracy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements—cross-fitting and loss correction—to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.
Introduction
Knowledge Distillation has become a cornerstone of efficient machine learning, allowing small "student" models to inherit the performance of large "teacher" ensembles or deep networks. A persistent mystery in the field is why students often achieve higher accuracy by following the teacher's "soft labels" than by training on the original "hard" ground-truth labels. This paper provides a rigorous theoretical foundation for this phenomenon by reframing KD through the lens of semiparametric inference. We identify two primary failure modes in standard KD—teacher overfitting and teacher underfitting—and propose statistically motivated solutions to address them.
A Semiparametric Perspective on KD
We model the distillation process as a semiparametric problem where:
- The Target: The optimal student model parameters.
- The Nuisance: The unknown true underlying class probabilities (Bayes probabilities).
- The Plug-in: The teacher model's predicted probabilities.
In this framework, the teacher serves as a proxy for the true data-generating distribution. Standard KD's success depends on how well the teacher estimates these nuisance probabilities. However, standard practices often reuse the same training data for both the teacher and the student, leading to a "data reuse trap" where the student inherits the teacher's noise and memorization rather than generalizable patterns.
Proposed Enhancements
To improve the robustness and accuracy of distillation, we introduce two key techniques:
1. Cross-fitting
Borrowed from modern statistical inference, cross-fitting involves partitioning the training data. The teacher is trained on one subset of data and provides soft labels for the student on a different, disjoint subset. This decoupling prevents the student from overfitting to the teacher's specific errors on the training set, effectively eliminating the complexity penalty associated with high-capacity teachers.
2. Loss Correction
To combat teacher underfitting (bias), we introduce a $\gamma$-corrected loss function. This approach adds a first-order correction term to the standard distillation loss using the actual ground-truth labels. Inspired by orthogonal machine learning, this makes the student's performance less sensitive to the teacher's approximation errors, allowing the student to potentially surpass the teacher's own accuracy.
Experimental Validation
We evaluated our enhancements across diverse benchmarks, including tabular datasets (FICO, Adult, Higgs, MAGIC, StumbleUpon) using Random Forests, and image data (CIFAR-10) using ResNets.
As shown in Figure 2, the combination of cross-fitting and loss correction (Enhanced KD Student) consistently outperforms both vanilla KD and models trained without distillation. On tabular data, we observed improvements of up to 4% AUC. On CIFAR-10, our methods improved student accuracy by up to 1.5% when the teacher was prone to overfitting.
Conclusion
By treating Knowledge Distillation as a semiparametric inference problem, we provide both a theoretical explanation for its success and practical tools to improve its performance. Cross-fitting and loss correction offer a robust path forward for model compression, ensuring that student models are not limited by the specific biases or overfitting tendencies of their teachers. Future work will explore the application of these semiparametric tools to self-distillation and augmented distillation using synthetic data.
Summary updated