Abstract
Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. In this work, we systematically compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English↔German, English↔Nepali and English↔Sinhala monolingual data, and evaluate them on NMT. In (semi-) supervised NMT, varying the pretraining objective leads to surprisingly small differences in the finetuned performance, whereas unsupervised NMT is much more sensitive to it. To understand these results, we thoroughly study the pretrained models using a series of probes and verify that they encode and use information in different ways. We conclude that finetuning on parallel data is mostly sensitive to few properties that are shared by most models, such as a strong decoder, in contrast to unsupervised NMT that also requires models with strong cross-lingual abilities.
Introduction
Neural Machine Translation (NMT) has traditionally relied on massive parallel corpora to achieve high performance. However, for low-resource languages, such data is often unavailable. Unsupervised pretraining on monolingual text has emerged as a powerful solution, typically utilizing Masked Language Modeling (MLM). While effective, MLM introduces a "pretraining-finetuning discrepancy": the model is trained on corrupted sentences with [MASK] tokens but is expected to translate full, natural sentences during inference. This study investigates whether "real-looking" noise—such as word shuffling or replacement—can provide a more effective pretraining signal for translation tasks.
Methodology: Beyond Masking
We systematically compare several noising strategies within a standard Transformer sequence-to-sequence architecture. These include:
- Masking (Baseline): Standard mBART-style masking of random tokens and spans.
- Replacing: Words are replaced with plausible alternatives from a small generator model, avoiding the use of explicit mask tokens.
- Replaced Token Detection (RTD): An auxiliary task where the encoder predicts whether each input token is original or a replacement.
- Shuffling: Local reordering of words to force the model to learn structural context without hiding information.
- Auxiliary Losses (eMLM): Additional encoder-specific tasks designed to ensure rich input representations.
Key Insights and Results
Supervised NMT is Pretraining-Agnostic
One of the most surprising findings is that supervised NMT is remarkably robust to the choice of pretraining objective. When even a small amount of parallel data is available, the specific noise used during pretraining has a negligible impact on final performance. Most methods improved over a random baseline by 5-8 BLEU points, but the variance between masking, shuffling, and replacement was often less than 1 BLEU point.
Unsupervised NMT is Fragile
In contrast, unsupervised NMT—which relies on iterative back-translation—is highly sensitive to the pretraining objective. In this setting, traditional masking remains the superior choice, outperforming replacement-based methods by up to 9 BLEU points. Probing suggests that masking encourages a "copying bias" that helps preserve entities (like names and dates) during the early, unstable stages of unsupervised learning, a feature that replacement-based models lack.
The Decoder as the Primary Driver
Through extensive ablation studies, we found that the primary benefit of pretraining for supervised NMT lies in the decoder's language modeling ability. When parts of the model are "reset" before fine-tuning, resetting the decoder leads to a much larger performance drop than resetting the encoder. This suggests that the parallel data used during fine-tuning is sufficient to "fix" cross-lingual alignment in the encoder, making the initial encoder representations less critical than the decoder's fluency.
Conclusion
While alternative pretraining objectives like word replacement and shuffling offer a more naturalistic training signal, they do not consistently outperform standard masking for NMT. Supervised models are largely indifferent to the encoder's specific pretraining noise, relying instead on the decoder's pretrained fluency. However, for unsupervised NMT, the structural clarity and copying bias provided by masking remain essential. Future research should explore hybrid objectives that combine the naturalism of replacement with the clear training signals of masking.
Summary updated