Unifying the Convergences in Multilingual Neural Machine Translation

Yi-Chong Huang, Xiaocheng Feng, Xinwei Geng, Bing Qin

Abstract

Although all-in-one-model multilingual neural machine translation (multilingual NMT) has achieved remarkable progress, the convergence inconsistency in the joint training is ignored, i.e., different language pairs reaching convergence in different epochs. This leads to the trained MNMT model over-fitting low-resource language translations while under-fitting high-resource ones. In this paper, we propose a novel training strategy named LSSD (Language-Specific Self-Distillation), which can alleviate the convergence inconsistency and help MNMT models achieve the best performance on each language pair simultaneously. Specifically, LSSD picks up language-specific best checkpoints for each language pair to teach the current model on the fly. Furthermore, we systematically explore three sample-level manipulations of knowledge transferring. Experimental results on three datasets show that LSSD obtains consistent improvements towards all language pairs and achieves the state-of-the-art.

Introduction

Multilingual Neural Machine Translation (MNMT) aims to train a single model capable of translating between multiple language pairs. While efficient, this joint training often leads to a "performance deficit" due to convergence inconsistency. Low-resource languages (LRLs) with limited data tend to converge quickly and overfit as training continues, whereas high-resource languages (HRLs) require significantly more epochs to reach peak performance. Standard training regimes typically select a single global checkpoint based on average performance, resulting in a model that is suboptimal for both LRLs (which have already degraded) and HRLs (which are still underfitting).

Methodology: Language-Specific Self-Distillation (LSSD)

To address this, we introduce Language-Specific Self-Distillation (LSSD). This strategy treats the current training model as a "student" that learns from "teachers"—specifically, the best historical checkpoints for each individual language pair.

The LSSD framework illustrating the distillation switch and the use of multiple language-specific teachers to guide the student model.

The Core Mechanism

LSSD maintains a repository of the best-performing checkpoints for every language pair. A Distillation Switch manages the training flow:

  • If the student outperforms the current teacher for a language pair, the teacher is updated, and distillation is paused.
  • If the student's performance begins to decline (indicating overfitting), the switch activates, and the teacher distills its knowledge back into the student to restore peak performance.

We explore three variants of this knowledge transfer:

  1. LSSD-WHOLE: Uniform distillation across all samples.
  2. LSSD-SELECTIVE: Distillation only on samples where the teacher exhibits higher confidence than the student.
  3. LSSD-ADAPTIVE: Dynamic weighting of distillation based on the performance gap between teacher and student.

Experimental Results

We evaluated LSSD on the TED-8-Diverse, TED-8-Related, and WMT datasets. The results demonstrate that LSSD effectively bridges the performance gap across diverse resource levels.

Validation loss curves comparing LSSD variants against the baseline. LSSD prevents the sharp rise in loss (overfitting) seen in low-resource languages while continuing to improve high-resource performance.

Key Findings

  • Mitigating Overfitting: As shown in the loss curves, LSSD prevents the performance degradation typically seen in LRLs (like Bosnian or Macedonian) during extended training.
  • Improving High-Resource Performance: By stabilizing LRLs, the model can train longer, allowing HRLs (like French or Korean) to reach deeper convergence. LSSD-Adaptive, in particular, showed superior robustness in Many-to-One scenarios.
  • Reduced Deficit: The proposed method reduced the Performance Deficit (measured by the Difference of Uniform Best metric) by an average of 57%.
  • Efficiency: While LSSD introduces a ~33% increase in training time due to teacher-student loss calculations, it incurs zero overhead during inference, as the teacher models are discarded post-training.

Conclusion

LSSD offers a robust solution to the convergence inconsistency problem in MNMT. By utilizing language-specific self-distillation, we can unify the training process, ensuring that the final model achieves state-of-the-art performance across all supported languages simultaneously without sacrificing inference efficiency.

Summary updated

Abstract

Although all-in-one-model multilingual neural machine translation (multilingual NMT) has achieved remarkable progress, the convergence inconsistency in the joint training is ignored, i.e., different language pairs reaching convergence in different epochs. This leads to the trained MNMT model over-fitting low-resource language translations while under-fitting high-resource ones. In this paper, we propose a novel training strategy named LSSD (Language-Specific Self-Distillation), which can alleviate the convergence inconsistency and help MNMT models achieve the best performance on each language pair simultaneously. Specifically, LSSD picks up language-specific best checkpoints for each language pair to teach the current model on the fly. Furthermore, we systematically explore three sample-level manipulations of knowledge transferring. Experimental results on three datasets show that LSSD obtains consistent improvements towards all language pairs and achieves the state-of-the-art.
BibTeX
@inproceedings{huang2022unifying,
  author = {Yi-Chong Huang and Xiaocheng Feng and Xinwei Geng and Bing Qin},
  title = {{Unifying the Convergences in Multilingual Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.458},
}
Figure