TL;DR
Monolingual data is a powerful tool for boosting multilingual machine translation (MMT), but its benefits are highly dependent on domain alignment and model capacity. While Backtranslation (BT) generally outperforms Denoising Autoencoding (DAE) at smaller scales, DAE becomes significantly more effective as models grow, eventually matching or even surpassing BT in low-resource settings at the 1.6B parameter scale.
What Problem Does This Paper Address?
Multilingual Machine Translation (MMT) aims to translate between many languages using a single model, which is particularly vital for low-resource languages that lack massive parallel corpora. To fill this data gap, researchers often supplement training with monolingual data using two main strategies: Backtranslation (BT) (generating synthetic parallel data) and Denoising Autoencoding (DAE) (learning to reconstruct corrupted sentences).
However, the research community has been divided. Some studies report universal gains from DAE, while others find it helps only in extreme low-resource cases or even harms performance. This paper sets out to resolve these contradictions by systematically investigating two often-overlooked factors: domain mismatch (e.g., training on Wikipedia but testing on News) and model scale (ranging from 90M to 1.6B parameters).
Methods
The authors conducted a large-scale study covering 100 translation directions (English $\leftrightarrow$ 50 languages). They compared a parallel-only baseline against models augmented with:
- Backtranslation (BT): The baseline model translates target-side monolingual data back into the source language to create synthetic pairs.
- Denoising Autoencoding (DAE): They tested two popular objectives:
- MASS: Masks a span in the input and requires the decoder to predict only that masked span.
- BART: Corrupts the input (shuffling, masking) and requires the decoder to reconstruct the entire original sentence.

Crucially, they varied the model size (90M, 370M, and 1.6B parameters) and the domain mixture of the monolingual data (Wikipedia-only vs. a mix of Web and News data).
Datasets
- Parallel Data: ML50, a diverse dataset covering 50 languages paired with English, categorized into high, medium, and low-resource groups.
- Monolingual Data: Wikipedia (for single-domain tests) and a mix of News Crawl and CC100 (Web) data.
- Evaluation Benchmarks: To test domain robustness, they evaluated on four distinct test sets:
- ML50 test: A mix of domains (TED, News, etc.).
- FLORES-200: Wikipedia domain.
- NTREX-128: News domain.
- TICO-19: Medical domain.
Key Insights and Ideas
- The "Brittleness" of Monolingual Data: Models are surprisingly sensitive to domain mismatches. If your monolingual data is from Wikipedia but your test set is News, the gains from BT and DAE can vanish or even turn negative.
- BT is More Sensitive than DAE: While BT often provides higher peaks in performance when domains match, it is more "brittle" than DAE when faced with out-of-domain data.
- Scale "Unblocks" DAE: This is perhaps the most significant finding. At small scales (90M), DAE (specifically MASS) often performs worse than the parallel-only baseline. As the model grows to 1.6B parameters, DAE's effectiveness skyrockets, bridging the gap with BT and even outperforming it in low-resource $XX \rightarrow EN$ directions.
- MASS > BART for Translation: Across almost all settings, the MASS objective consistently outperformed BART. The authors speculate that BART’s requirement to reconstruct the entire sentence (including unmasked tokens) biases the model toward simple copying rather than deep semantic denoising.
Key Results
- Average Gains: At the 370M parameter scale, BT improved the baseline by +1.4 BLEU, while DAE (MASS) improved it by +0.7 BLEU.
- Domain Impact: Gains were significantly higher when the test set matched the monolingual data domain. For example, improvements on FLORES (Wiki) were much larger than on the mixed-domain ML50 test set.
- Scaling Success: In out-of-domain settings, the 1.6B parameter MASS model transitioned from being detrimental at small scales to becoming competitive with BT.
Limitations and Future Work
The authors acknowledge several boundaries to their study:
- English-Centricity: The study focused on English $\leftrightarrow$ XX directions. Results might differ for many-to-many multilingual models without English as a pivot.
- Parameter Ceiling: While 1.6B is large for traditional NMT, it is small compared to modern Large Language Models (LLMs). Whether DAE continues to pull ahead of BT at the 10B+ scale remains an open question.
- Methodological Scope: The study did not include newer techniques like contrastive learning or specialized prompting for LLMs, focusing instead on the most widely adopted joint-training methods.
Future work could explore these scaling laws in fully many-to-many settings and investigate if the "copying bias" in BART can be mitigated to make it more effective for translation tasks.
Summary updated