Introduction
Multilingual Neural Machine Translation (MNMT) aims to build a single model capable of translating between multiple language pairs. While this approach benefits from cross-lingual transfer, it introduces a significant challenge known as convergence inconsistency. In a standard MNMT setup, different language pairs reach their peak performance at different stages of training. High-resource languages (HRLs) typically require extensive training to converge, whereas low-resource languages (LRLs) often converge quickly and subsequently begin to overfit.
When a "best" model is selected based on the average validation loss across all languages, the resulting checkpoint is often a compromise: it may be under-fitted for HRLs while already overfitting on LRLs. This gap—the difference between the global best checkpoint and the potential peak performance of each individual language pair—is termed the Performance Deficit.
Language-Specific Self-Distillation (LSSD)
To address this, we propose Language-Specific Self-Distillation (LSSD). Unlike traditional distillation that relies on external, pre-trained teacher models, LSSD utilizes the model's own historical best checkpoints for each specific language pair as "teachers."
The Distillation Mechanism
The core of LSSD is the Distillation Switch. For every language pair $l$, the system monitors the validation loss. If the current model's performance on pair $l$ degrades compared to its historical best, the distillation switch is activated. The historical best checkpoint then serves as a teacher, providing soft targets to guide the current student model back toward its peak performance. Conversely, if the current model achieves a new best validation loss for pair $l$, the switch is turned off, and the historical best is updated with the current model parameters.
Implementation Variants
The authors explored three strategies for applying distillation at the sample level:
- LSSD-Whole: Applies distillation to all samples within a language pair uniformly.
- LSSD-Selective: Only distills samples where the teacher's likelihood is higher than the student's, ensuring the student only learns from the teacher's superior states.
- LSSD-Adaptive: Dynamically weights the distillation loss based on the performance ratio between the teacher and the student.
Experimental Evaluation
LSSD was evaluated on three benchmarks: TED-8-Diverse, TED-8-Related, and a larger-scale WMT dataset. The results demonstrate that LSSD consistently outperforms standard multilingual baselines across all language pairs.
Key findings include:
- Significant BLEU Gains: LSSD achieved an average improvement of +1.77 BLEU on the TED-8-Diverse Many-to-One (M2O) task, with gains as high as +2.37 BLEU for specific pairs like English → Macedonian.
- Reducing Performance Deficit: The method successfully reduced the "Performance Deficit" by an average of 57%, bringing the final global model much closer to the per-language theoretical maximums.
- Robustness: LSSD outperformed "Multi-Distill," a state-of-the-art baseline that uses separate bilingual models as teachers, proving that self-distillation is both effective and more efficient than maintaining external teachers.
Conclusion
OmniKnight introduces a self-contained strategy to harmonize the learning rates of diverse language pairs in MNMT. By "remembering" its own best performance for each task, the model prevents overfitting in low-resource scenarios while allowing high-resource learning to continue. While LSSD introduces some training overhead (approximately 33% increase in training time on TED datasets), it requires no additional parameters at inference time and significantly improves the overall quality of multilingual translation.
Summary updated