Abstract
We characterize the statistical efficiency of knowledge transfer through $n$ samples from a teacher to a probabilistic student classifier with input space $\mathcal S$ over labels $\mathcal A$. We show that privileged information at three progressive levels accelerates the transfer. At the first level, only samples with hard labels are known, via which the maximum likelihood estimator attains the minimax rate $\sqrt{{|{\mathcal S}||{\mathcal A}|}/{n}}$. The second level has the teacher probabilities of sampled labels available in addition, which turns out to boost the convergence rate lower bound to ${{|{\mathcal S}||{\mathcal A}|}/{n}}$. However, under this second data acquisition protocol, minimizing a naive adaptation of the cross-entropy loss results in an asymptotically biased student. We overcome this limitation and achieve the fundamental limit by using a novel empirical variant of the squared error logit loss. The third level further equips the student with the soft labels (complete logits) on ${\mathcal A}$ given every sampled input, thereby provably enables the student to enjoy a rate ${|{\mathcal S}|}/{n}$ free of $|{\mathcal A}|$. We find any Kullback-Leibler divergence minimizer to be optimal in the last case. Numerical simulations distinguish the four learners and corroborate our theory.
Introduction
Knowledge Distillation (KD) has become a cornerstone of modern AI, enabling the compression of massive models into efficient versions suitable for deployment. Despite its empirical success, a rigorous understanding of the "fundamental limits" of KD has been lacking. This paper addresses this gap by characterizing the statistical efficiency of knowledge transfer across three distinct levels of teacher disclosure, establishing the theoretical "speed limits" at which a student can learn from a teacher.
The Information Ladder
The researchers define three levels of teacher information, each providing a different degree of "privileged information" to the student:
- Hard Labels: The student only observes the final classification (e.g., "This is a cat"). This corresponds to standard supervised learning.
- Partial Soft Labels: The student receives the hard label and the teacher’s specific probability (confidence) for that label.
- Full Soft Labels (Logits): The student receives the teacher’s entire probability distribution across all possible classes for every sampled input.
Methodology and Loss Functions
The study employs a minimax analysis framework within finite (tabular) domains to determine the optimal convergence rates. A significant finding of this work is the "Cross-Entropy Trap." While Cross-Entropy (CE) is the industry standard for KD, the authors prove that a naive adaptation of CE fails when only partial soft labels are available, leading to an asymptotically biased student. To resolve this, they introduce a novel Squared Error Logit (SEL) loss, which aligns the student's internal logits with the teacher's probabilities without the bias inherent in CE for partial data.
Key Results
The theoretical findings demonstrate that learning speed scales predictably with the richness of teacher disclosure. Moving from hard labels to partial soft labels provides a quadratic reduction in error. Crucially, when full logits are provided, the learning rate becomes independent of the number of classes ($|\mathcal{A}|$), explaining why KD is exceptionally effective in tasks with large label spaces, such as language modeling.
As shown in the table above, the performance improves significantly as more information is transferred. For partial soft labels, the table highlights the failure of $CE_{ptl}$ (achieving only $\Omega(1)$) compared to the optimal rate achieved by $SEL_{ptl}$.
Discussion and Limitations
While this work establishes fundamental limits, it focuses on the finite domain (tabular) setting. In practical applications, neural networks utilize function approximation rather than tables. While the authors provide preliminary discussions on generalizing these results to log-linear models, extending this theory to deep neural architectures remains a vital area for future research. Other future directions include investigating the impact of student initialization and the use of unnormalized logits.
Summary updated