Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation

Songming Zhang, Yunlong Liang, Shuaibo Wang, Wenjuan Han, Jian Liu, Jinan Xu, Yufeng Chen

Abstract

Knowledge distillation (KD) is a cornerstone technique for model compression in neural machine translation (NMT). However, the specific nature of the "knowledge" being transferred remains poorly understood. In this work, we empirically demonstrate that the primary source of knowledge in KD resides in the teacher's top-1 predictions rather than the broader probability distribution. Based on this finding, we identify two limitations in vanilla word-level KD: a lack of focus on top-1 information and significant overlap between teacher predictions and ground-truth labels. To address these, we propose Top-1 Information Enhanced Knowledge Distillation (TIE-KD), which utilizes a hierarchical ranking loss and an iterative training procedure. Experiments across WMT benchmarks show that TIE-KD significantly outperforms vanilla KD and sequence-level KD, providing gains of up to +1.11 BLEU and +4.80 COMET points.

Introduction

In the pursuit of efficient Neural Machine Translation (NMT), researchers often employ Knowledge Distillation (KD) to transfer the performance of large, computationally expensive "teacher" models into smaller, deployable "student" models. While effective, the mechanism of KD is often treated as a black box. The prevailing theory suggests that "dark knowledge"—the relative probabilities assigned to incorrect candidate words—is the key to student improvement.

This paper challenges that assumption, revealing that the teacher's top-1 prediction is the most critical component. By refocusing KD on this specific information, we can overcome the inherent inefficiencies of standard word-level distillation.

The Mystery: Where Does the Knowledge Hide?

To locate the source of knowledge, the authors conducted ablation studies on the teacher's output distribution. They found that when the teacher's top-1 prediction is removed or masked, the student's performance drops precipitously. Conversely, removing the "correlation information" (the distribution of all other tokens) has almost no negative impact on the final translation quality.

This finding suggests a bridge between word-level KD (mimicking distributions) and sequence-level KD (mimicking teacher-generated sequences): both are essentially trying to align the student's top-1 choice with the teacher's best guess. Furthermore, the study suggests that much of the extra distribution data in vanilla KD acts more as noise than useful signal.

Challenges in Vanilla KD

The authors identify two primary issues that hinder standard word-level KD:

  1. Lack of Focus: Standard loss functions like Kullback-Leibler (KL) Divergence treat the entire probability distribution equally. They do not explicitly prioritize the teacher's top-1 prediction, which is the most informative element.
  2. Information Overlap: In many cases, the teacher's top-1 prediction is identical to the ground-truth (gold) token. When this happens, the student learns nothing from the teacher that it wouldn't have learned from standard supervised training, limiting the potential for improvement.

Proposed Method: TIE-KD

To resolve these issues, the Top-1 Information Enhanced Knowledge Distillation (TIE-KD) framework introduces two novel components:

Hierarchical Ranking Loss ($L_{hr}$)

Instead of merely minimizing the distance between two probability distributions, $L_{hr}$ explicitly enforces a ranking. It ensures that the student model ranks the teacher’s top-1 prediction as its own top-1. This hierarchical approach forces the student to prioritize the most critical information provided by the teacher.

Iterative KD Procedure

To address the "overlap" problem, the authors implement an iterative training loop. The student generates its own predictions, which are then used as inputs for the next round of distillation. Because these student-generated sequences often differ from the ground truth, the teacher is forced to provide guidance on "synthetic" data where its top-1 prediction is likely to offer new, non-redundant information.

Experimental Results

The method was evaluated on three major benchmarks: WMT’14 English-German (En-De), WMT’14 English-French (En-Fr), and WMT’16 English-Romanian (En-Ro). Using a Transformer-big teacher and a Transformer-base student, TIE-KD achieved the following:

  • BLEU Improvements: TIE-KD boosted student scores by +1.04 (En-De), +0.60 (En-Fr), and +1.11 (En-Ro) over the baseline.
  • COMET Gains: On the English-Romanian task, the method showed a remarkable +4.80 improvement in COMET scores, a metric that correlates more closely with human perception of quality.
  • Robustness: TIE-KD demonstrated higher generalizability across different teacher-student capacity gaps compared to existing techniques, sometimes allowing the student to approach or even match the teacher's performance.

Conclusion

By shifting the focus of Knowledge Distillation from the entire probability distribution to the teacher's top-1 predictions, TIE-KD provides a more efficient and effective way to train NMT models. This research not only provides a new state-of-the-art method for word-level distillation but also clarifies the underlying mechanics of how students learn from teachers in machine translation tasks.

Summary updated

Abstract

Knowledge distillation (KD) is a promising technique for model compression in neural machine translation. However, where the knowledge hides in KD is still not clear, which may hinder the development of KD. In this work, we first unravel this mystery from an empirical perspective and show that the knowledge comes from the top-1 predictions of teachers, which also helps us build a potential connection between word- and sequence-level KD. Further, we point out two inherent issues in vanilla word-level KD based on this finding. Firstly, the current objective of KD spreads its focus to whole distributions to learn the knowledge, yet lacks special treatment on the most crucial top-1 information. Secondly, the knowledge is largely covered by the golden information due to the fact that most top-1 predictions of teachers overlap with ground-truth tokens, which further restricts the potential of KD. To address these issues, we propose a novel method named \textbf{T}op-1 \textbf{I}nformation \textbf{E}nhanced \textbf{K}nowledge \textbf{D}istillation (TIE-KD). Specifically, we design a hierarchical ranking loss to enforce the learning of the top-1 information from the teacher. Additionally, we develop an iterative KD procedure to infuse more additional knowledge by distilling on the data without ground-truth targets. Experiments on WMT'14 English-German, WMT'14 English-French and WMT'16 English-Romanian demonstrate that our method can respectively boost Transformer$_{base}$ students by +1.04, +0.60 and +1.11 BLEU scores and significantly outperform the vanilla word-level KD baseline. Besides, our method shows higher generalizability on different teacher-student capacity gaps than existing KD techniques.
BibTeX
@inproceedings{zhang2023towards,
  author = {Songming Zhang and Yunlong Liang and Shuaibo Wang and Wenjuan Han and Jian Liu and Jinan Xu and Yufeng Chen},
  title = {{Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=lJ8zLVt1qc},
}
Figure