When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale

Christos Baziotis, Biao Zhang, Alexandra Birch, Barry Haddow

Abstract

Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods of including monolingual data. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 translation directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.

Introduction

Multilingual Machine Translation (MMT) aims to translate between numerous languages using a single unified model. This approach is particularly transformative for low-resource languages where parallel training data is scarce. To bridge the gap, researchers frequently augment training with monolingual data through two primary techniques: Backtranslation (BT), which generates synthetic parallel data, and Denoising Autoencoding (DAE), which teaches the model language structure by reconstructing corrupted text.

Despite their popularity, reports on the effectiveness of these methods are often contradictory. This research systematically investigates the factors that determine when and how monolingual data actually helps, focusing on the critical roles of domain alignment and model scale.

Methodology and Data

The study utilizes the Transformer architecture across three scales: Base (90M parameters), Big (370M), and XL (1.6B). The primary methods for integrating monolingual data are:

  • Backtranslation (BT): Generating synthetic source sentences from target-side monolingual data.
  • MASS (Masked Sequence to Sequence): A DAE objective where the model predicts a masked span of text.
  • BART: A DAE objective where the model reconstructs an entire sentence from a corrupted version.
The MASS denoising objective masks a span in the encoder and predicts it in the decoder.

The experiments utilize the ML50 dataset, covering 50 languages with varying resource levels. Evaluation is performed across multiple benchmarks to test domain robustness: FLORES-200 (Wikipedia), NTREX-128 (News), TICO-19 (Medical), and the ML50 Test Set (Mixed/TED).

Key Insights: Domain and Scale

The "Brittleness" of Backtranslation

A major finding is the sensitivity of BT to domain mismatches. If the monolingual data (e.g., Wikipedia) does not align with the test domain (e.g., Medical), BT can degrade performance compared to a baseline trained only on parallel data. Increasing the diversity of monolingual data (mixing News, Web, and Wiki) is shown to significantly improve BT's robustness.

Scale Unlocks Denoising Potential

The effectiveness of DAE objectives, specifically MASS, is heavily dependent on model scale. At the 90M parameter scale, MASS often fails to beat the parallel-only baseline. However, as models grow to 1.6B parameters, MASS performance improves rapidly, eventually converging with or even surpassing BT, particularly in low-resource settings. MASS consistently outperformed BART, likely because BART's requirement to reconstruct unmasked tokens encourages simple copying rather than deep linguistic learning.

Results and Analysis

At medium scales (370M parameters), BT remains the most reliable method, provided there is reasonable domain alignment. However, at the 1.6B scale, the gap between BT and MASS closes significantly.

Performance gains (Δ-BLEU) of MASS and BT over a parallel-only baseline across FLORES (Wiki) and NTREX (News) benchmarks.

As shown in the results across different benchmarks, the gains from monolingual data are most pronounced for low-resource languages (right side of the x-axis in the plots). In ideal conditions, adding monolingual data at the 1.6B scale yielded improvements of up to +5 BLEU points for these languages.

Conclusion and Future Work

This study clarifies the conditions under which monolingual data benefits MMT. While BT is a strong baseline, its performance is contingent on domain similarity. DAE objectives like MASS require significant model scale to be effective but offer competitive or superior performance in low-resource scenarios as they scale.

Future research should explore these dynamics in "Many-to-Many" models that do not rely on English as a pivot and investigate whether DAE continues to gain an advantage at the 10B+ parameter scale common in modern Large Language Models.

Summary updated

Abstract

Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods of including monolingual data. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 translation directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.
BibTeX
@inproceedings{baziotis2024when,
  author = {Christos Baziotis and Biao Zhang and Alexandra Birch and Barry Haddow},
  title = {{When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TgplOs0weI},
}
Figure