TL;DR
Masking spans of text (as in mBART and MASS) creates a discrepancy between pretraining and inference, but replacing masks with more natural noise like word substitutions or shuffling doesn't meaningfully improve supervised neural machine translation (NMT). Through extensive probing, the authors discover that supervised NMT mainly needs a strong pretrained decoder (acting as a language model), whereas unsupervised NMT is uniquely reliant on masking because it teaches the model to copy invariant tokens like dates and named entities.
What Problem Does This Paper Address?
Standard sequence-to-sequence pretraining for NMT relies heavily on Masked Language Modeling (MLM), where tokens or spans are masked with special symbols ([MASK]) and reconstructed by the decoder. However, this creates a pretraining–finetuning discrepancy: during pretraining, encoders process corrupted, masked text, but during finetuning and inference, they receive complete, natural sentences.
This paper investigates whether alternative pretraining objectives that corrupt sentences without masking—such as word substitution using an auxiliary generator or local shuffling—can eliminate this discrepancy and improve translation. Crucially, the authors seek to understand which specific components (encoder, decoder, or embeddings) and representational properties actually drive translation gains.
Methods
The authors evaluate several denoising objectives for a multilingual sequence-to-sequence Transformer (base configuration):
- Masking (Baseline): Replacing random tokens or spans with
[MASK]tokens (following mBART). - Masking + eMLM: Augmenting sequence-to-sequence masking with an explicit auxiliary MLM classification loss directly on top of the encoder.
- Replacing (ELECTRA-style): Masked tokens are replaced by samples drawn from a small, jointly trained cross-lingual masked language model generator. The authors test both untied generators and tied generators (where encoder and generator share parameters).
- Replacement + Replaced Token Detection (RTD): A discriminator head is added to the encoder to classify whether each input token is original or synthetic.
- Shuffling: Words are locally permuted within a window $k$ without adding mask tokens, discouraging the cross-attention layers from learning trivial monotonic alignments.
- Probing and Ablation Framework: To diagnose what the models learn, the authors introduce:
- Component ablations: Selectively resetting encoder, decoder, cross-attention, or embedding weights while freezing the rest to see which sub-networks matter most.
- Diagnostic probes: Measuring token denoising perplexity in the encoder, decoder predictive entropy, cross-lingual sentence retrieval, and sensitivity to blocked encoder states.
Datasets
The authors evaluate on low-resource translation across three diverse language pairs:
- English $\leftrightarrow$ German (En $\leftrightarrow$ De): Monolingual data from WMT News Crawl (190M English, 270M German sentences). Low-resource parallel data from WMT News Commentary v13 (~275k sentences). Dev/Test:
newstest2017,newstest2018,newstest2019. Parallel sentence retrieval evaluated on the Tatoeba test set. - English $\leftrightarrow$ Nepali (En $\leftrightarrow$ Ne) & English $\leftrightarrow$ Sinhala (En $\leftrightarrow$ Si): Monolingual data (5M sentences each from Common Crawl/Wikipedia) and parallel datasets (~600k sentences from Bible, OpenSubtitles, Paracrawl) from the FLORES benchmark (Guzmán et al., 2019).
Key Insights and Ideas
- Supervised NMT is remarkably indifferent to encoder noise: Whether corrupting inputs via masking, replacing words, or simply shuffling word orders, supervised finetuning arrives at nearly identical BLEU scores. Parallel data easily bridges cross-lingual discrepancies in the encoder.
- Pretraining mostly benefits the decoder: Component freezing ablations show that resetting the pretrained decoder causes a significantly larger drop in downstream translation performance than resetting the encoder. Pretraining acts primarily as target-language model training for the decoder.
- Unsupervised NMT tells the opposite story: In unsupervised NMT, masking outperforms generator replacements and crushes shuffling (by up to 8.5 BLEU). Why? Unsupervised back-translation requires early translation hooks. Mask-based models learn a strong bias to copy uncorrupted tokens (achieving zero entropy on unmasked inputs), allowing entities, numbers, and proper nouns to pass through cleanly and bootstrap iterative back-translation. Replacement-based models learn that any word could be fake, making them hesitant to copy.
- NLU breakthroughs don't directly transfer to NMT: Adding RTD heads (which excel in models like ELECTRA) actually harmed translation quality and degraded the encoder's cross-lingual sentence retrieval capabilities, particularly in the deeper encoder layers.
Key Results
- Supervised NMT: All pretraining methods substantially beat random initialization (e.g., En$\rightarrow$De WMT18 jumps from 26.2 BLEU random to 33.3–34.2 BLEU across pretraining schemes). "Replace + Tied + Shuffling" reached the highest En$\rightarrow$De scores (34.0 on WMT18; 31.1 on WMT19), but simple token shuffling alone reached an impressive 31.6 BLEU on WMT18.
- Semi-Supervised (Back-translation): Differences between methods narrowed even further (ranging within ~0.8 BLEU of each other on En$\rightarrow$De), indicating that back-translation synthetic data largely neutralizes minor pretraining differences.
- Unsupervised NMT (En$\rightarrow$De WMT18):
- Masking (35%): 25.6 BLEU
- Replace (35%): 23.3 BLEU
- Shuffle ($k=5$): 17.1 BLEU (a steep drop illustrating that decoder fluency without good cross-lingual encoder alignment cannot sustain unsupervised translation).
- Component Ablations (De$\rightarrow$En WMT19): Re-initializing the encoder alone retains 23.2 BLEU, whereas re-initializing the decoder drops performance to 20.7 BLEU, and re-initializing the tied embeddings drops performance drastically to 11.8 BLEU.
Limitations and Future Work
- Model Capacity: The study was limited to Transformer-base architectures (6 layers, 512 hidden dimension) due to computational constraints across many pretraining runs; results may shift at massive scale (e.g., mBART-50).
- Unsupervised Scope: Unsupervised translation experiments were limited strictly to English–German, as fully unsupervised translation on distant, low-resource pairs like English–Nepali and English–Sinhala completely failed to yield meaningful BLEU scores.
- Hyperparameter Exploration: The RTD loss weight was held fixed ($\lambda = 25$) without exhaustive hyperparameter tuning, which may have contributed to its sub-par cross-lingual performance.
- Anomalous Encoder Probing: In sentence retrieval probes, standard masking exhibited an unexplained drop in retrieval accuracy across middle encoder layers before recovering at the final layer, an architectural artifact left for future work to explain.
Summary updated