When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale

Christos Baziotis, Biao Zhang, Alexandra Birch, Barry Haddow

TL;DR

Monolingual data is a powerful tool for boosting multilingual machine translation (MMT), but its benefits are highly dependent on domain alignment and model capacity. While Backtranslation (BT) generally outperforms Denoising Autoencoding (DAE) at smaller scales, DAE becomes significantly more effective as models grow, eventually matching or even surpassing BT in low-resource settings at the 1.6B parameter scale.

What Problem Does This Paper Address?

Multilingual Machine Translation (MMT) aims to translate between many languages using a single model, which is particularly vital for low-resource languages that lack massive parallel corpora. To fill this data gap, researchers often supplement training with monolingual data using two main strategies: Backtranslation (BT) (generating synthetic parallel data) and Denoising Autoencoding (DAE) (learning to reconstruct corrupted sentences).

However, the research community has been divided. Some studies report universal gains from DAE, while others find it helps only in extreme low-resource cases or even harms performance. This paper sets out to resolve these contradictions by systematically investigating two often-overlooked factors: domain mismatch (e.g., training on Wikipedia but testing on News) and model scale (ranging from 90M to 1.6B parameters).

Methods

The authors conducted a large-scale study covering 100 translation directions (English $\leftrightarrow$ 50 languages). They compared a parallel-only baseline against models augmented with:

  1. Backtranslation (BT): The baseline model translates target-side monolingual data back into the source language to create synthetic pairs.
  2. Denoising Autoencoding (DAE): They tested two popular objectives:
  3. MASS: Masks a span in the input and requires the decoder to predict only that masked span.
  4. BART: Corrupts the input (shuffling, masking) and requires the decoder to reconstruct the entire original sentence.

Illustration of the MASS objective Illustration of the BART objective

Crucially, they varied the model size (90M, 370M, and 1.6B parameters) and the domain mixture of the monolingual data (Wikipedia-only vs. a mix of Web and News data).

Datasets

  • Parallel Data: ML50, a diverse dataset covering 50 languages paired with English, categorized into high, medium, and low-resource groups.
  • Monolingual Data: Wikipedia (for single-domain tests) and a mix of News Crawl and CC100 (Web) data.
  • Evaluation Benchmarks: To test domain robustness, they evaluated on four distinct test sets:
  • ML50 test: A mix of domains (TED, News, etc.).
  • FLORES-200: Wikipedia domain.
  • NTREX-128: News domain.
  • TICO-19: Medical domain.

Key Insights and Ideas

  • The "Brittleness" of Monolingual Data: Models are surprisingly sensitive to domain mismatches. If your monolingual data is from Wikipedia but your test set is News, the gains from BT and DAE can vanish or even turn negative.
  • BT is More Sensitive than DAE: While BT often provides higher peaks in performance when domains match, it is more "brittle" than DAE when faced with out-of-domain data.
  • Scale "Unblocks" DAE: This is perhaps the most significant finding. At small scales (90M), DAE (specifically MASS) often performs worse than the parallel-only baseline. As the model grows to 1.6B parameters, DAE's effectiveness skyrockets, bridging the gap with BT and even outperforming it in low-resource $XX \rightarrow EN$ directions.
  • MASS > BART for Translation: Across almost all settings, the MASS objective consistently outperformed BART. The authors speculate that BART’s requirement to reconstruct the entire sentence (including unmasked tokens) biases the model toward simple copying rather than deep semantic denoising.

Key Results

  • Average Gains: At the 370M parameter scale, BT improved the baseline by +1.4 BLEU, while DAE (MASS) improved it by +0.7 BLEU.
  • Domain Impact: Gains were significantly higher when the test set matched the monolingual data domain. For example, improvements on FLORES (Wiki) were much larger than on the mixed-domain ML50 test set.
  • Scaling Success: In out-of-domain settings, the 1.6B parameter MASS model transitioned from being detrimental at small scales to becoming competitive with BT.
Figure 4: BLEU differences between each model and the baseline (red dotted line) on FLORES and NTREX.

Limitations and Future Work

The authors acknowledge several boundaries to their study:

  • English-Centricity: The study focused on English $\leftrightarrow$ XX directions. Results might differ for many-to-many multilingual models without English as a pivot.
  • Parameter Ceiling: While 1.6B is large for traditional NMT, it is small compared to modern Large Language Models (LLMs). Whether DAE continues to pull ahead of BT at the 10B+ scale remains an open question.
  • Methodological Scope: The study did not include newer techniques like contrastive learning or specialized prompting for LLMs, focusing instead on the most widely adopted joint-training methods.

Future work could explore these scaling laws in fully many-to-many settings and investigate if the "copying bias" in BART can be mitigated to make it more effective for translation tasks.

Summary updated

Abstract

Multilingual machine translation (MMT), trained on a mixture of parallel and monolingual data, is key for improving translation in low-resource language pairs. However, the literature offers conflicting results on the performance of different methods of including monolingual data. To resolve this, we examine how denoising autoencoding (DAE) and backtranslation (BT) impact MMT under different data conditions and model scales. Unlike prior studies, we use a realistic dataset of 100 translation directions and consider many domain combinations of monolingual and test data. We find that monolingual data generally helps MMT, but models are surprisingly brittle to domain mismatches, especially at smaller model scales. BT is beneficial when the parallel, monolingual, and test data sources are similar but can be detrimental otherwise, while DAE is less effective than previously reported. Next, we analyze the impact of scale (from 90M to 1.6B parameters) and find it is important for both methods, particularly DAE. As scale increases, DAE transitions from underperforming the parallel-only baseline at 90M to converging with BT performance at 1.6B, and even surpassing it in low-resource. These results offer new insights into how to best use monolingual data in MMT.
BibTeX
@inproceedings{baziotis2024when,
  author = {Christos Baziotis and Biao Zhang and Alexandra Birch and Barry Haddow},
  title = {{When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TgplOs0weI},
}
Figure