This paper introduces MARGE (Multilingual Autoencoder that Retrieves and Generates), a novel pre-training objective for language models that moves beyond the dominant "fill-in-the-blank" approach of Masked Language Modeling (MLM). Instead of reconstructing masked words, MARGE learns by retrieving related documents across multiple languages and using them to paraphrase a target document. This effectively unifies translation, summarization, and retrieval into a single unsupervised task.
TL;DR
MARGE is a viable alternative to Masked Language Modeling for pre-training large-scale multilingual models. By learning to reconstruct a document from a retrieved set of related documents in many languages, it achieves remarkable zero-shot performance on translation and summarization without ever seeing a single parallel sentence during training.
Background and Motivation
For years, the "gold standard" for pre-training models like BERT or RoBERTa has been Masked Language Modeling (MLM): hide some words, and ask the model to predict them. While highly effective, MLM is a local task focused on word-level reconstruction.
The authors of MARGE argue that MLM doesn't explicitly capture higher-level linguistic tasks like paraphrasing, document-level reasoning, or cross-lingual mapping. Furthermore, learning multilingual representations usually requires either shared vocabularies or expensive "bitext" (manually translated parallel sentences). MARGE seeks a more holistic, unsupervised objective that naturally forces the model to learn how information relates across different documents and different languages.
Methods: Pre-training via Paraphrasing
The core idea of MARGE is pre-training via paraphrasing. The process follows two main steps:
- Retrieval: Given a target document $x$, a retrieval model scores the relevance of other "evidence" documents $z_j$ from a massive multilingual corpus. This relevance is calculated as the cosine similarity between document embeddings.
- Reconstruction: A sequence-to-sequence (seq2seq) Transformer model then tries to generate the original target document $x$, conditioned on the retrieved evidence documents $z_{1..M}$.
A key innovation is the biased cross-attention mechanism. During reconstruction, the decoder's attention is weighted by the relevance scores from the retrieval model. This forces the model to "listen" more to the most relevant evidence.
To solve the "chicken-and-egg" problem—needing a good model to retrieve relevant documents, but needing relevant documents to train the model—the authors found that even a randomly initialized model can start the process by picking up on simple lexical overlaps. They then iteratively rebuild the document index every 10,000 updates to refine the retrieval.
Datasets
- Pre-training: The model was trained on a multilingual version of the CC-NEWS corpus and subsequently on Wikipedia, covering 26 different languages.
- Evaluation: The authors tested MARGE on a wide battery of benchmarks including BUCC2018 and Tatoeba (Retrieval), IWSLT2017 and WMT19 (Translation), MLSum (Summarization), PAWS-X (Paraphrasing), and MLQA (Question Answering).
Key Insights
- Paraphrasing as a Super-Task: Reconstructing a document from related ones "noisily captures" translation, summarization, and information retrieval, making the model inherently versatile.
- Emergent Translation: Perhaps the most striking insight is that MARGE learns to translate zero-shot. By retrieving a French news article about an event to reconstruct an English article about the same event, the model learns the mapping between languages without explicit parallel data.
- Linguistic Affinities: The retrieval model doesn't just find random documents; it learns to cluster languages based on linguistic and geographical relationships. As seen in the heatmap below, the model naturally identifies clusters like Germanic, Romance, and Slavic languages.
- Synthesis over Memorization: Analysis shows the model isn't just memorizing text; it synthesizes facts from the evidence documents, often producing outputs that are more than just literal translations.
Key Results
- Zero-Shot Translation: MARGE achieved a BLEU score of 35.8 for German-to-English document translation with zero task-specific training or parallel data.
- Cross-Lingual Retrieval: On the BUCC task, MARGE outperformed other unsupervised models by nearly 10 points.
- State-of-the-Art Paraphrasing: It set a new state-of-the-art for zero-shot transfer on the PAWS-X paraphrase detection dataset.
- Competitive Summarization: In multilingual summarization (MLSum), MARGE outperformed extractive mBERT and other strong baselines, showing it can effectively condense information across languages.
Limitations and Future Work
- Metadata Reliance: The current implementation uses simple metadata (like publication dates in news or inter-language links in Wikipedia) to make the initial retrieval search efficient. Scaling this to "unstructured" web data would require more complex approximate nearest neighbor searches.
- Compute Intensity: Learning the retrieval model requires large batch sizes, which necessitates significant computational resources and model-parallel training for larger versions.
In conclusion, MARGE represents a significant shift in NLP pre-training, proving that a retrieval-based paraphrasing objective can create models that are not only competitive with MLM-based ones but significantly more capable in zero-shot multilingual scenarios.
Summary updated