When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale

Christos Baziotis, Biao Zhang, Alexandra Birch, Barry Haddow

Abstract

Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods of including monolingual data. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 translation directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.

Introduction

Multilingual Machine Translation (MMT) has emerged as a powerful paradigm for enabling translation across dozens of languages within a single model. This is particularly critical for low-resource languages where parallel corpora (translated text pairs) are scarce. To supplement these limited resources, researchers typically turn to monolingual data—text available in only one language—using two primary techniques: Backtranslation (BT) and Denoising Autoencoding (DAE).

Despite their widespread use, the effectiveness of these methods is often debated. Some studies suggest DAE provides universal improvements, while others report negligible or even negative impacts. This paper addresses this inconsistency by systematically investigating the roles of domain alignment and model scale in determining when and how monolingual data truly benefits MMT.

Methodology

The authors conducted a large-scale study covering 100 translation directions. They established a "parallel-only" baseline using the ML50 dataset and compared it against models augmented with BT and two variants of DAE: MASS and BART.

  • MASS (Masked Sequence-to-Sequence): The model is trained to predict a masked span of text in the decoder, given the context from the encoder.
  • BART: The model reconstructs an entire sentence from a corrupted version.
The MASS denoising objective: masking a span in the encoder and predicting it in the decoder.

The study varied model capacity across three scales: Base (90M), Big (370M), and XL (1.6B) parameters. Furthermore, they tested the models against diverse evaluation benchmarks, including FLORES-200 (Wikipedia domain), NTREX-128 (News), and TICO-19 (Medical), to assess robustness across different data distributions.

Key Insights and Results

The Scale-Dependency of DAE

A major finding is that DAE is significantly more dependent on model scale than BT. At the 90M parameter scale, DAE methods often fail to outperform the parallel-only baseline. However, as the model size increases to 1.6B parameters, DAE (specifically the MASS objective) shows dramatic improvements, eventually matching or even exceeding BT performance in low-resource "Many-to-English" scenarios.

Domain Sensitivity: The "BT Diva" Effect

While BT often provides the highest peak gains, it is highly sensitive to domain alignment. If the monolingual data used for BT (e.g., Wikipedia) does not match the domain of the test set (e.g., News), the performance gains can evaporate or turn negative. DAE, while generally providing lower peak gains at medium scales, tends to be slightly more robust to these mismatches as scale increases.

Comparative performance of MASS and BT across different benchmarks (FLORES, NTREX) and translation directions.

Objective Matters: MASS vs. BART

The MASS objective consistently outperformed BART for translation tasks. The authors suggest that BART's objective of full-sentence reconstruction may encourage the model to rely on simple "copying" behaviors. In contrast, MASS forces the decoder to extract more meaningful representations from the encoder, which better prepares the model for the actual task of translation.

Conclusion and Practical Recommendations

This research clarifies the conditions under which monolingual data is most effective. For practitioners, the choice of method should be guided by available compute and data domain:

  1. Small-to-Medium Models (<1B parameters): Backtranslation remains the most effective choice, provided the monolingual data domain closely matches the target application domain.
  2. Large Models (1.6B+ parameters): DAE (specifically MASS) becomes a highly competitive and computationally cheaper alternative to BT, especially for low-resource languages.
  3. Data Diversity: Regardless of the method, using a diverse mixture of monolingual data (e.g., combining Web, News, and Wiki) is essential for building models that are robust to domain shifts.

Future work remains to be done in scaling these findings to the massive parameter counts of modern Large Language Models (LLMs) and exploring non-English-centric translation directions.

Summary updated

Abstract

Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods of including monolingual data. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 translation directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.
BibTeX
@inproceedings{baziotis2024when,
  author = {Christos Baziotis and Biao Zhang and Alexandra Birch and Barry Haddow},
  title = {{When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TgplOs0weI},
}
Figure