Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation

Songming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen, Wenjuan Han, Jian Liu, Jinan Xu

Authors: Songming Zhang, Yunlong Liang, Shuaibo Wang, Yufeng Chen, Wenjuan Han, Jian Liu, Jinan Xu.
Source: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.

Introduction

Neural Machine Translation (NMT) models have achieved high accuracy but are often too large for efficient deployment. Knowledge Distillation (KD) is a standard technique for compressing these large "teacher" models into smaller "student" models. While word-level KD is widely used, the underlying mechanism—often attributed to "dark knowledge" or the semantic relationships between non-target tokens—has remained poorly understood in the context of NMT. This paper investigates the source of KD's effectiveness and proposes a method to bridge the gap between word-level and sequence-level distillation.

The Myth of Dark Knowledge in NMT

A central contribution of this work is the empirical investigation into what constitutes "dark knowledge" in NMT. Traditionally, it was believed that the entire probability distribution, including correlations between negative candidates, was essential. However, the authors' ablation studies reveal a different reality.

Ablation study of different information components in word-level KD showing that top-1 information is the primary driver of performance.

As illustrated in the figure above, removing correlation information (b) has a negligible impact on performance compared to the vanilla baseline (a). In contrast, removing the top-1 prediction information (c) causes the distillation process to degrade significantly, performing similarly to training without KD (d). This suggests that in NMT, the teacher's top-1 prediction is the primary carrier of knowledge.

Methodology: TIE-KD

Based on these insights, the authors propose Top-1 Information Enhanced Knowledge Distillation (TIE-KD). TIE-KD focuses on explicitly reinforcing the teacher's top-1 preference through two main mechanisms:

  1. Hierarchical Ranking Loss ($L_{hr}$): Standard KD uses Kullback-Leibler (KL) divergence, which treats the entire probability distribution equally. TIE-KD adds a specific ranking loss that forces the student to prioritize the teacher’s top-1 prediction above all other candidates. This ensures the student captures the most "crucial" piece of knowledge explicitly.
  2. Iterative Knowledge Distillation: In many NMT datasets, the teacher's top-1 prediction matches the ground truth over 90% of the time. This overlap limits the "extra" knowledge the student can gain. TIE-KD employs an iterative process where the student's own predictions are used as inputs, forcing the teacher to provide guidance on "non-golden" data and exposing its unique decision-making logic.

A Unified Theory of KD

The authors provide a theoretical proof showing that word-level and sequence-level KD are essentially performing the same task: encouraging the student to match the teacher's top-1 preference. The primary difference lies in the "context" used during training—whether it is ground-truth (word-level) or model-generated (sequence-level). This insight bridges the conceptual divide between these two common distillation approaches.

Experimental Results

The researchers evaluated TIE-KD on WMT’14 En-De, En-Fr, and WMT’16 En-Ro benchmarks using a Transformer-big teacher and a Transformer-base student.

TIE-KD consistently outperformed both vanilla word-level KD and sequence-level KD (Seq-KD). For instance, on WMT’14 En-De, TIE-KD achieved a +1.04 BLEU improvement over the baseline and +0.43 BLEU over vanilla Word-KD. These gains were also reflected in COMET scores, which are more aligned with human judgment, showing improvements of up to +4.80 points on En-Ro. Furthermore, TIE-KD demonstrated robustness to large capacity gaps, maintaining strong performance even with deeper teacher models.

Conclusion

This research clarifies that the effectiveness of KD in NMT stems primarily from the teacher's top-1 predictions rather than complex inter-token correlations. By unifying word-level and sequence-level KD perspectives, the authors provide a more efficient distillation framework. While the iterative procedure increases training time, the resulting performance gains and theoretical clarity offer a significant step forward for NMT model compression.

Summary updated

BibTeX
@inproceedings{zhang2023towards,
  author = {Songming Zhang and Yunlong Liang and Shuaibo Wang and Yufeng Chen and Wenjuan Han and Jian Liu and Jinan Xu},
  title = {{Towards Understanding and Improving Knowledge Distillation for Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=lJ8zLVt1qc},
}
Figure