Abstract
Knowledge distillation (KD) is a cornerstone technique for model compression in neural machine translation (NMT). However, the specific nature of the "knowledge" being transferred remains poorly understood. In this work, we empirically demonstrate that the primary source of knowledge in KD resides in the teacher's top-1 predictions rather than the broader probability distribution. Based on this finding, we identify two limitations in vanilla word-level KD: a lack of focus on top-1 information and significant overlap between teacher predictions and ground-truth labels. To address these, we propose Top-1 Information Enhanced Knowledge Distillation (TIE-KD), which utilizes a hierarchical ranking loss and an iterative training procedure. Experiments across WMT benchmarks show that TIE-KD significantly outperforms vanilla KD and sequence-level KD, providing gains of up to +1.11 BLEU and +4.80 COMET points.
Introduction
In the pursuit of efficient Neural Machine Translation (NMT), researchers often employ Knowledge Distillation (KD) to transfer the performance of large, computationally expensive "teacher" models into smaller, deployable "student" models. While effective, the mechanism of KD is often treated as a black box. The prevailing theory suggests that "dark knowledge"—the relative probabilities assigned to incorrect candidate words—is the key to student improvement.
This paper challenges that assumption, revealing that the teacher's top-1 prediction is the most critical component. By refocusing KD on this specific information, we can overcome the inherent inefficiencies of standard word-level distillation.
The Mystery: Where Does the Knowledge Hide?
To locate the source of knowledge, the authors conducted ablation studies on the teacher's output distribution. They found that when the teacher's top-1 prediction is removed or masked, the student's performance drops precipitously. Conversely, removing the "correlation information" (the distribution of all other tokens) has almost no negative impact on the final translation quality.
This finding suggests a bridge between word-level KD (mimicking distributions) and sequence-level KD (mimicking teacher-generated sequences): both are essentially trying to align the student's top-1 choice with the teacher's best guess. Furthermore, the study suggests that much of the extra distribution data in vanilla KD acts more as noise than useful signal.
Challenges in Vanilla KD
The authors identify two primary issues that hinder standard word-level KD:
- Lack of Focus: Standard loss functions like Kullback-Leibler (KL) Divergence treat the entire probability distribution equally. They do not explicitly prioritize the teacher's top-1 prediction, which is the most informative element.
- Information Overlap: In many cases, the teacher's top-1 prediction is identical to the ground-truth (gold) token. When this happens, the student learns nothing from the teacher that it wouldn't have learned from standard supervised training, limiting the potential for improvement.
Proposed Method: TIE-KD
To resolve these issues, the Top-1 Information Enhanced Knowledge Distillation (TIE-KD) framework introduces two novel components:
Hierarchical Ranking Loss ($L_{hr}$)
Instead of merely minimizing the distance between two probability distributions, $L_{hr}$ explicitly enforces a ranking. It ensures that the student model ranks the teacher’s top-1 prediction as its own top-1. This hierarchical approach forces the student to prioritize the most critical information provided by the teacher.
Iterative KD Procedure
To address the "overlap" problem, the authors implement an iterative training loop. The student generates its own predictions, which are then used as inputs for the next round of distillation. Because these student-generated sequences often differ from the ground truth, the teacher is forced to provide guidance on "synthetic" data where its top-1 prediction is likely to offer new, non-redundant information.
Experimental Results
The method was evaluated on three major benchmarks: WMT’14 English-German (En-De), WMT’14 English-French (En-Fr), and WMT’16 English-Romanian (En-Ro). Using a Transformer-big teacher and a Transformer-base student, TIE-KD achieved the following:
- BLEU Improvements: TIE-KD boosted student scores by +1.04 (En-De), +0.60 (En-Fr), and +1.11 (En-Ro) over the baseline.
- COMET Gains: On the English-Romanian task, the method showed a remarkable +4.80 improvement in COMET scores, a metric that correlates more closely with human perception of quality.
- Robustness: TIE-KD demonstrated higher generalizability across different teacher-student capacity gaps compared to existing techniques, sometimes allowing the student to approach or even match the teacher's performance.
Conclusion
By shifting the focus of Knowledge Distillation from the entire probability distribution to the teacher's top-1 predictions, TIE-KD provides a more efficient and effective way to train NMT models. This research not only provides a new state-of-the-art method for word-level distillation but also clarifies the underlying mechanics of how students learn from teachers in machine translation tasks.
Summary updated