Exploring Unsupervised Pretraining Objectives for Machine Translation

Christos Baziotis, Ivan Titov, Alexandra Birch, Barry Haddow

Abstract

Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. In this work, we systematically compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English↔German, English↔Nepali and English↔Sinhala monolingual data, and evaluate them on NMT. In (semi-) supervised NMT, varying the pretraining objective leads to surprisingly small differences in the finetuned performance, whereas unsupervised NMT is much more sensitive to it. To understand these results, we thoroughly study the pretrained models using a series of probes and verify that they encode and use information in different ways. We conclude that finetuning on parallel data is mostly sensitive to few properties that are shared by most models, such as a strong decoder, in contrast to unsupervised NMT that also requires models with strong cross-lingual abilities.

Introduction

Neural Machine Translation (NMT) has traditionally relied on massive parallel corpora to achieve high performance. However, for low-resource languages, such data is often unavailable. Unsupervised pretraining on monolingual text has emerged as a powerful solution, typically utilizing Masked Language Modeling (MLM). While effective, MLM introduces a "pretraining-finetuning discrepancy": the model is trained on corrupted sentences with [MASK] tokens but is expected to translate full, natural sentences during inference. This study investigates whether "real-looking" noise—such as word shuffling or replacement—can provide a more effective pretraining signal for translation tasks.

Methodology: Beyond Masking

We systematically compare several noising strategies within a standard Transformer sequence-to-sequence architecture. These include:

  • Masking (Baseline): Standard mBART-style masking of random tokens and spans.
  • Replacing: Words are replaced with plausible alternatives from a small generator model, avoiding the use of explicit mask tokens.
  • Replaced Token Detection (RTD): An auxiliary task where the encoder predicts whether each input token is original or a replacement.
  • Shuffling: Local reordering of words to force the model to learn structural context without hiding information.
  • Auxiliary Losses (eMLM): Additional encoder-specific tasks designed to ensure rich input representations.
Figure 1: Overview of the noise detection and reconstruction framework, comparing shuffling and replacement strategies against the standard encoder-decoder flow.

Key Insights and Results

Supervised NMT is Pretraining-Agnostic

One of the most surprising findings is that supervised NMT is remarkably robust to the choice of pretraining objective. When even a small amount of parallel data is available, the specific noise used during pretraining has a negligible impact on final performance. Most methods improved over a random baseline by 5-8 BLEU points, but the variance between masking, shuffling, and replacement was often less than 1 BLEU point.

Unsupervised NMT is Fragile

In contrast, unsupervised NMT—which relies on iterative back-translation—is highly sensitive to the pretraining objective. In this setting, traditional masking remains the superior choice, outperforming replacement-based methods by up to 9 BLEU points. Probing suggests that masking encourages a "copying bias" that helps preserve entities (like names and dates) during the early, unstable stages of unsupervised learning, a feature that replacement-based models lack.

The Decoder as the Primary Driver

Through extensive ablation studies, we found that the primary benefit of pretraining for supervised NMT lies in the decoder's language modeling ability. When parts of the model are "reset" before fine-tuning, resetting the decoder leads to a much larger performance drop than resetting the encoder. This suggests that the parallel data used during fine-tuning is sufficient to "fix" cross-lingual alignment in the encoder, making the initial encoder representations less critical than the decoder's fluency.

Figure 3: Heatmap of ablation studies showing BLEU scores when resetting specific components (embeddings, decoder, encoder). The sharper drops in the '- dec' column highlight the decoder's critical role.

Conclusion

While alternative pretraining objectives like word replacement and shuffling offer a more naturalistic training signal, they do not consistently outperform standard masking for NMT. Supervised models are largely indifferent to the encoder's specific pretraining noise, relying instead on the decoder's pretrained fluency. However, for unsupervised NMT, the structural clarity and copying bias provided by masking remain essential. Future research should explore hybrid objectives that combine the naturalism of replacement with the clear training signals of masking.

Summary updated

Abstract

Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT), by drastically reducing the need for large parallel data. Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. In this work, we systematically compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. We pretrain models with different methods on English$\leftrightarrow$German, English$\leftrightarrow$Nepali and English$\leftrightarrow$Sinhala monolingual data, and evaluate them on NMT. In (semi-) supervised NMT, varying the pretraining objective leads to surprisingly small differences in the finetuned performance, whereas unsupervised NMT is much more sensitive to it. To understand these results, we thoroughly study the pretrained models using a series of probes and verify that they encode and use information in different ways. We conclude that finetuning on parallel data is mostly sensitive to few properties that are shared by most models, such as a strong decoder, in contrast to unsupervised NMT that also requires models with strong cross-lingual abilities.
BibTeX
@inproceedings{baziotis2021exploring,
  author = {Christos Baziotis and Ivan Titov and Alexandra Birch and Barry Haddow},
  title = {{Exploring Unsupervised Pretraining Objectives for Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Tmi5G1SXYWs},
}
Figure