OmniKnight: Multilingual Neural Machine Translation with Language-Specific Self-Distillation

Yi-Chong Huang, Xiaocheng Feng, Xinwei Geng, Bing Qin

Introduction

Multilingual Neural Machine Translation (MNMT) aims to build a single model capable of translating between multiple language pairs. While this approach benefits from cross-lingual transfer, it introduces a significant challenge known as convergence inconsistency. In a standard MNMT setup, different language pairs reach their peak performance at different stages of training. High-resource languages (HRLs) typically require extensive training to converge, whereas low-resource languages (LRLs) often converge quickly and subsequently begin to overfit.

Validation loss curves for different language pairs, illustrating the asynchronous convergence points that lead to performance deficits in standard MNMT.

When a "best" model is selected based on the average validation loss across all languages, the resulting checkpoint is often a compromise: it may be under-fitted for HRLs while already overfitting on LRLs. This gap—the difference between the global best checkpoint and the potential peak performance of each individual language pair—is termed the Performance Deficit.

Language-Specific Self-Distillation (LSSD)

To address this, we propose Language-Specific Self-Distillation (LSSD). Unlike traditional distillation that relies on external, pre-trained teacher models, LSSD utilizes the model's own historical best checkpoints for each specific language pair as "teachers."

The LSSD framework showing the distillation switch mechanism and the process of updating language-specific teachers during training.

The Distillation Mechanism

The core of LSSD is the Distillation Switch. For every language pair $l$, the system monitors the validation loss. If the current model's performance on pair $l$ degrades compared to its historical best, the distillation switch is activated. The historical best checkpoint then serves as a teacher, providing soft targets to guide the current student model back toward its peak performance. Conversely, if the current model achieves a new best validation loss for pair $l$, the switch is turned off, and the historical best is updated with the current model parameters.

Implementation Variants

The authors explored three strategies for applying distillation at the sample level:

  1. LSSD-Whole: Applies distillation to all samples within a language pair uniformly.
  2. LSSD-Selective: Only distills samples where the teacher's likelihood is higher than the student's, ensuring the student only learns from the teacher's superior states.
  3. LSSD-Adaptive: Dynamically weights the distillation loss based on the performance ratio between the teacher and the student.

Experimental Evaluation

LSSD was evaluated on three benchmarks: TED-8-Diverse, TED-8-Related, and a larger-scale WMT dataset. The results demonstrate that LSSD consistently outperforms standard multilingual baselines across all language pairs.

Key findings include:

  • Significant BLEU Gains: LSSD achieved an average improvement of +1.77 BLEU on the TED-8-Diverse Many-to-One (M2O) task, with gains as high as +2.37 BLEU for specific pairs like English → Macedonian.
  • Reducing Performance Deficit: The method successfully reduced the "Performance Deficit" by an average of 57%, bringing the final global model much closer to the per-language theoretical maximums.
  • Robustness: LSSD outperformed "Multi-Distill," a state-of-the-art baseline that uses separate bilingual models as teachers, proving that self-distillation is both effective and more efficient than maintaining external teachers.

Conclusion

OmniKnight introduces a self-contained strategy to harmonize the learning rates of diverse language pairs in MNMT. By "remembering" its own best performance for each task, the model prevents overfitting in low-resource scenarios while allowing high-resource learning to continue. While LSSD introduces some training overhead (approximately 33% increase in training time on TED datasets), it requires no additional parameters at inference time and significantly improves the overall quality of multilingual translation.

Summary updated

BibTeX
@inproceedings{huang2022omniknight,
  author = {Yi-Chong Huang and Xiaocheng Feng and Xinwei Geng and Bing Qin},
  title = {{OmniKnight: Multilingual Neural Machine Translation with Language-Specific Self-Distillation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.458},
}
Figure