Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation

Biao Zhang, Philip Williams, Ivan Titov, Rico Sennrich

Abstract

Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.

Introduction

Massively multilingual neural machine translation (NMT) aims to consolidate hundreds of language pairs into a single model. While this approach promises benefits like cross-lingual knowledge transfer and simplified deployment, current models often fall short of specialized bilingual baselines. This research identifies two primary obstacles: a capacity bottleneck that limits the model's ability to represent diverse linguistic structures, and the off-target issue, where the model fails to produce the correct target language during zero-shot translation.

Addressing the Off-Target Issue

In zero-shot scenarios—translating between languages never explicitly paired during training—models frequently produce text in the wrong language. As shown below, the model might copy the source text or produce a hybrid of the source and target languages instead of the intended translation.

Examples of off-target translation where the model fails to produce the correct target language in zero-shot settings.

Proposed Methods

To overcome these hurdles, the authors propose a two-pronged strategy:

1. Increasing Architectural Capacity

To handle the complexity of 100+ languages, the architecture was scaled and specialized:

  • Deep Transformers: Scaling the standard 6-layer Transformer architecture up to 24 layers.
  • Language-aware Layer Normalization (LALN): Conditioning layer normalization parameters on the specific target language to provide language-specific "hooks."
  • Language-aware Linear Transformation (LALT): Inserting a language-specific linear layer between the encoder and decoder to improve mapping to target-specific representations.

2. Random Online Backtranslation (ROBT)

To mitigate the off-target problem, Random Online Backtranslation (ROBT) was introduced. During fine-tuning, the model back-translates target sentences into randomly selected intermediate languages. This creates "pseudo-parallel" data on the fly, forcing the model to learn the translation path between arbitrary language pairs and ensuring it adheres to the requested target language.

Experimental Setup: OPUS-100

The study utilizes OPUS-100, a new English-centric multilingual corpus covering 100 languages with approximately 55 million sentence pairs. Evaluation focuses on both supervised directions and 15 sampled zero-shot language pairs (e.g., Arabic-Chinese, French-German).

Key Results

The combination of increased capacity and ROBT yielded significant improvements:

  • Supervised Performance: The 24-layer model with LALN/LALT effectively closed the gap with bilingual models, sometimes even outperforming them by small margins.
  • Zero-Shot Breakthrough: ROBT provided a massive boost to zero-shot quality, increasing performance by approximately 10 BLEU points and significantly improving language accuracy (ACC_zero), as detailed in the table below.
Zero-shot translation performance metrics showing the significant impact of ROBT on BLEU and language accuracy.

Conclusion

This work demonstrates that the limitations of massively multilingual NMT can be addressed through targeted architectural enhancements and robust training strategies like ROBT. While LALT adds language-specific parameters that may scale poorly to thousands of languages, the current approach provides a strong foundation for high-quality, zero-shot-capable multilingual translation. Future work will explore more parameter-efficient scaling and non-English-centric datasets.

Summary updated

Abstract

Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by ~10 BLEU, approaching conventional pivot-based methods.
BibTeX
@inproceedings{zhang2020improving,
  author = {Biao Zhang and Philip Williams and Ivan Titov and Rico Sennrich},
  title = {{Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=a6pFUofUQK},
}
Figure