TL;DR
This paper investigates the claim that Knowledge Distillation (KD) is a form of Label Smoothing (LS). Through empirical analysis on text classification tasks, the authors demonstrate that KD and LS have fundamentally different effects on model confidence, with KD generally reducing uncertainty while LS increases it, thus refuting the equivalence.
What Problem Does This Paper Address?
Knowledge Distillation (KD) is a popular technique for training smaller, more efficient models by transferring knowledge from a larger "teacher" model. Recently, some research has proposed that KD is essentially a form of regularization, drawing parallels to Label Smoothing (LS). Label Smoothing, on the other hand, is a well-established regularization technique designed to prevent overfitting by making models less confident in their predictions. The core problem this paper addresses is the validity of this claimed equivalence between KD and LS. If KD is indeed just a form of LS, it implies a different understanding of how knowledge is transferred and how models are regularized. This paper aims to clarify this by empirically comparing the two methods.
Methods
The authors re-examine the claimed equivalence by comparing the predictive confidences of models trained with KD and LS. They analyze the posterior entropy of model predictions, which serves as a measure of confidence.
- Knowledge Distillation (KD): The standard KD setup involves training a student model using soft targets (probability distributions) provided by a pre-trained teacher model. The paper considers standard KD (large teacher to small student), self-KD (teacher and student are the same architecture), and reverse KD (small teacher to large student).
- Label Smoothing (LS): LS is applied by replacing the one-hot ground truth labels with a smoothed distribution, typically a mixture of the true label and a uniform distribution.
- Empirical Comparison: The core of the methodology involves training models using both KD and LS on various text classification tasks and then comparing their posterior entropies. A higher posterior entropy indicates lower confidence, while a lower entropy indicates higher confidence.
Datasets
The experiments were conducted on four text classification tasks from the GLUE benchmark:
- MRPC (Microsoft Research Paraphrase Corpus): Determines if two sentences are paraphrases.
- QNLI (Question Natural Language Inference): Checks if a sentence answers a given question.
- SST-2 (Stanford Sentiment Treebank): Classifies movie reviews as positive or negative.
- MNLI (Multi-Genre Natural Language Inference): A three-way classification task (entailment, contradiction, neutral) for premise-hypothesis pairs.
The paper also details the dataset sizes and splits used for training, validation, and testing in an appendix.
Key Insights and Ideas
- Opposite Effects on Confidence: The most striking finding is that KD and LS generally have opposite effects on model confidence. LS consistently increases posterior entropy (reduces confidence), as expected. In contrast, KD, particularly standard and self-KD, typically reduces posterior entropy (increases confidence), indicating a more confident student model.
- Teacher's Influence is Crucial: The confidence of the student model in KD is shown to be a direct function of the teacher model's confidence. This supports the classical view of KD as knowledge transfer, where the student inherits not just task knowledge but also the teacher's learned confidence.
- "Teacher-Free KD" Re-evaluation: The paper offers a perspective on "teacher-free KD" methods, suggesting that their performance might be better explained by the inherent regularization effects of using smoothed targets rather than a direct equivalence to LS.
- Reverse KD Nuance: While standard KD reduces entropy, reverse KD (distilling from a smaller to a larger model) can sometimes increase entropy, but still to a lesser extent than LS. This highlights the role of model size and architecture in KD's effect on confidence.
Key Results
- Posterior Entropy: On the four GLUE tasks, LS consistently increased posterior entropy over standard Empirical Risk Minimization (ERM) by a significant margin (around 3-6x). KD, on the other hand, generally decreased posterior entropy. For instance, on SST-2, standard KD reduced entropy from 0.1101 (ERM) to 0.0107.
- Accuracy: In most cases, KD variants achieved better test set accuracy than LS. For example, on MRPC, standard KD achieved 82.0% accuracy compared to LS's 78.4%.
- Teacher-Student Relationship: The experiments showed that the student's confidence in KD is strongly correlated with the teacher's confidence. When a BERT (larger) teacher was used, the DistilBERT (smaller) student generally exhibited lower entropy (higher confidence) than when distilled from a DistilBERT teacher.
Limitations and Future Work
- Single Random Seed: The authors acknowledge that due to the complexity of their experiments (four tasks, large hyperparameter grids), they ran experiments with a single random seed. While sufficient for demonstrating the core entropy findings, more reliable accuracy figures would require averaging over multiple seeds.
- Future Work: The paper suggests that future research should focus on better understanding distillation and its potential for improving model utility. The authors hope their work will inspire further efforts to clarify the mechanisms of distillation and prevent misinterpretations. They also hint at exploring distillation at higher temperatures (Appendix A.3), which did not qualitatively change their main findings.
Summary updated