Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich

Abstract: Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

2020.

This paper addresses the "capacity bottleneck" and "off-target" translation issues inherent in massively multilingual neural machine translation (NMT). By introducing language-aware layers and a novel Random Online Backtranslation (ROBT) finetuning method, the authors significantly narrow the performance gap between universal models and specialized bilingual ones, while boosting zero-shot translation quality by approximately 10 BLEU points.

The Challenges of Multilingual NMT

Massively multilingual NMT—training a single model to handle dozens or hundreds of languages—is a major goal in the field because it simplifies deployment and aids low-resource languages through transfer learning. However, it faces two primary hurdles:

  1. The Capacity Bottleneck: As more languages are added, the model’s fixed parameters are stretched thin, leading to degraded performance compared to specialized bilingual models.
  2. The Off-Target Issue in Zero-Shot Translation: When translating between language pairs not seen together during training (e.g., French $\rightarrow$ German in an English-centric model), models often fail to decouple meaning from the target language signal. This results in "off-target" outputs where the model either copies the source text or translates into the wrong language (typically English).
Examples of the off-target issue where the model fails to translate into the correct target language in zero-shot scenarios.

Proposed Methods

The authors propose a two-part solution focusing on architectural depth and targeted finetuning.

1. Increasing Model Capacity

Instead of simply increasing model width, the researchers focus on depth and language-specific conditioning:

  • Deep Transformers: The architecture is scaled up to 24 layers using specialized initialization to maintain training stability.
  • Language-Aware Layer Normalization (LALN): Standard Layer Norm is adapted to allow gain and bias parameters to be specific to the target language.
  • Language-Aware Linear Transformation (LALT): A language-specific linear layer is inserted between the encoder and decoder, acting as a flexible bridge that helps the model adapt internal representations to specific target language requirements.

2. Fixing Zero-Shot with ROBT

To solve the off-target problem, the authors introduce Random Online Backtranslation (ROBT). During a finetuning phase, the model takes a target-side sentence and translates it back into a randomly sampled intermediate language from the 100 available. It then learns to translate that synthetic source back into the original target language. This forces the model to respect the target language token and learn how to translate into it from any arbitrary starting point, effectively simulating the zero-shot environment during training.

Datasets: OPUS-100

The authors introduced and released OPUS-100, a large-scale, English-centric multilingual corpus covering 100 languages and 55M sentence pairs. Unlike previous proprietary datasets, OPUS-100 is open-source, providing a standardized benchmark for the research community. Evaluation covered one-to-many, many-to-many, and zero-shot directions.

Key Results and Insights

  • Narrowing the Gap: On one-to-many tasks, the 24-layer model with LALN/LALT achieved 29.60 BLEU, outperforming baseline bilingual models (20.90 BLEU) and significantly beating the standard multilingual baseline (24.64 BLEU).
  • Zero-Shot Breakthrough: In zero-shot scenarios, the baseline model was nearly useless, often copying the source or translating to English (Accuracy ~36%). With ROBT, the translation-language accuracy jumped to ~86%, and BLEU scores improved by roughly 10 points.
  • Efficiency and Convergence: Surprisingly, ROBT does not require extensive extra training. As shown in the performance curves, the model "clicks" and learns to stay on-target within just a few thousand finetuning steps.
Zero-shot BLEU performance over training steps, showing the rapid improvement ("click") once finetuning begins.

Conclusion

The paper demonstrates that poor zero-shot performance is primarily a failure of the model to respect target language constraints rather than a lack of underlying translation capability. While language-aware layers provide the necessary capacity for massive multilinguality, ROBT provides the essential signal to utilize that capacity correctly in zero-shot settings. Future work may explore lighter versions of LALT to improve scaling efficiency further.

Summary updated

BibTeX
@inproceedings{zhang2020improving,
  author = {Biao Zhang and Philip Williams and Ivan Titov and Rico Sennrich},
  title = {{Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=a6pFUofUQK},
}
Figure