Towards Higher Pareto Frontier in Multilingual Machine Translation

Yichong Huang, Xiaocheng Feng, Xinwei Geng, Baohang Li, Bing Qin

TL;DR

Multilingual machine translation often forces a zero-sum game where improving performance for low-resource languages comes at the expense of high-resource ones. This paper introduces Pareto Mutual Distillation (Pareto-MD), a framework that trains two models with opposing language biases to learn from each other’s strengths, effectively pushing the entire performance frontier outward rather than just making trade-offs.

What Problem Does This Paper Address?

In Multilingual Neural Machine Translation (MNMT), a single model is trained to handle multiple language pairs. Because training data is typically "long-tailed"—with a few languages having massive corpora (High-Resource Languages, or HRLs) and many having very little (Low-Resource Languages, or LRLs)—researchers face a Pareto optimization problem.

If you train the model normally, it excels at HRLs but ignores LRLs. If you use "balancing" strategies (like oversampling LRL data), the model improves on LRLs but its performance on HRLs degrades. This creates a Pareto frontier: a curve representing the best possible trade-offs where you can't improve one objective without hurting another. The authors argue that existing methods merely move the model to different points along this frontier. Their goal is to push the entire frontier outward, achieving better results for all languages simultaneously.

Figure 1: Multilingual performance frontier shifts out- wards. X-axis and Y-axis indicate the performance of Low-Resource Languages and High-Resource Lan- guages, respectively. Existing methods reflect a trade- off on the Pareto frontier (i.e., the gray curve). Our work aims to push the original Pareto frontier i.e., the blue dotted curve. To this effect, we ameliorate each individ- ual model’s shortcoming while retaining their strengths, e.g., moving right the solution A to A1 and moving up the solution B to B1, via our Pareto Mutual Distillation.

Methods

The authors propose Pareto Mutual Distillation (Pareto-MD). Instead of training one model, they train two multilingual models ($\theta_1$ and $\theta_2$) simultaneously:

  1. Diverse Sampling: $\theta_1$ is trained with a distribution favoring HRLs, while $\theta_2$ uses a distribution favoring LRLs.
  2. Mutual Distillation: At every training step, the models act as both student and teacher to one another. $\theta_1$ learns how to handle LRLs from $\theta_2$, and $\theta_2$ learns how to handle HRLs from $\theta_1$.
Figure 2: Illustration of Pareto-MD, using different sampling distributions to train two models. At each step, both models additionally mimic the output of each other via knowledge distillation. The distillation learning of each model is weighted by language-specific distillation weights αirℓs deduced with specific strategies.

The most advanced version, AUTO-PMD, introduces a dynamic way to manage this "teaching." Since different language pairs require different amounts of guidance at different times, AUTO-PMD uses a trial-and-error approach (modeled as a Markov Chain) to automatically search for optimal, language-specific distillation weights ($\alpha$) during training.

The AUTO-PMD update process: making trials with increased, decreased, or unchanged weights to find the optimal action for each language.

Datasets

The researchers evaluated their method on two major benchmarks:

  • WMT-6: A dataset involving 3 LRLs (Estonian, Romanian, Turkish) and 3 HRLs (French, German, Chinese) translated to/from English, totaling about 5 million sentence pairs.
  • TED-8-Diverse: A smaller but more varied dataset with 4 LRLs (Bosnian, Marathi, Hindi, Macedonian) and 4 HRLs (Greek, Bulgarian, French, Korean) translated to/from English.

Key Insights and Ideas

  • Synergy over Trade-off: The core "aha" moment is that models trained on different data distributions capture different subsets of relevant features. By sharing these features through distillation, the models can compensate for their inherent biases.
  • LRLs Need More Guidance: The authors found that LRLs generally favor higher distillation weights. This makes sense intuitively: LRLs are prone to overfitting, so they benefit more from the "soft" labels provided by a teacher model that has seen more diverse data.
  • Dynamic Weighting is Crucial: Fixed distillation weights are sub-optimal because the "knowledge gap" between the two models changes as they evolve during the training process. AUTO-PMD’s ability to adapt these weights on the fly is a significant contributor to its success.

Key Results

Pareto-MD consistently outperformed state-of-the-art baselines (like temperature-based sampling and LSSD) across both datasets:

  • Frontier Pushing: As shown in the visualization below, the models don't just slide along the trade-off curve; they move significantly to the top-right, indicating simultaneous gains in HRL and LRL performance.
  • BLEU Gains: On the WMT-6 Many-to-One task, AUTO-PMD improved performance by up to +2.46 BLEU over standard temperature sampling and +1.22 BLEU over the previous state-of-the-art.
  • Robustness: The method proved effective in both Many-to-One (multiple languages to English) and One-to-Many (English to multiple languages) scenarios.
The performance frontier for One-to-Many translation, showing AUTO-PMD (dark red) pushing well beyond the baseline (gray).

Limitations and Future Work

  • Computational Cost: The primary drawback is that Pareto-MD doubles the training cost because two models must be trained simultaneously. However, the authors note that this does not affect inference speed, as you only need to deploy one of the resulting models.
  • Scaling Up: The current work focuses on training two models. The authors suggest that future research could explore training even more models collaboratively to cover more complex Pareto frontiers.
  • LLM Integration: A promising next step is applying this framework to the training of massive Large Language Models (LLMs), where data imbalance across languages is an even more significant hurdle.

Summary updated

Abstract

Multilingual neural machine translation has witnessed remarkable progress in recent years. However, the long-tailed distribution of multilingual corpora poses a challenge of Pareto optimization, i.e., optimizing for some languages may come at the cost of degrading the performance of others. Existing balancing training strategies are equivalent to a series of Pareto optimal solutions, which trade off on a Pareto frontier. In this work, we propose a new training framework, Pareto Mutual Distillation (Pareto-MD), towards pushing the Pareto frontier outwards rather than making trade-offs. Specifically, Pareto-MD collaboratively trains two Pareto optimal solutions that favor different languages and allows them to learn from the strengths of each other via knowledge distillation. Furthermore, we introduce a novel strategy to enable stronger communication between Pareto optimal solutions and broaden the applicability of our approach. Experimental results on the widely-used WMT and TED datasets show that our method significantly pushes the Pareto frontier and outperforms baselines by up to +2.46 BLEU.
BibTeX
@inproceedings{huang2023towards,
  author = {Yichong Huang and Xiaocheng Feng and Xinwei Geng and Baohang Li and Bing Qin},
  title = {{Towards Higher Pareto Frontier in Multilingual Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.211},
}
Figure