Abstract
We introduce MARGE, a pre-trained sequence-to-sequence model learned with an unsupervised multi-lingual multi-document paraphrasing objective. MARGE provides an alternative to the dominant masked language modeling paradigm, where we self-supervise the reconstruction of target text by retrieving a set of related texts (in many languages) and conditioning on them to maximize the likelihood of generating the original. We show it is possible to jointly learn to do retrieval and reconstruction, given only a random initialization. The objective noisily captures aspects of paraphrase, translation, multi-document summarization, and information retrieval, allowing for strong zero-shot performance on several tasks. For example, with no additional task-specific training we achieve BLEU scores of up to 35.8 for document translation. We further show that fine-tuning gives strong performance on a range of discriminative and generative tasks in many languages, making MARGE the most generally applicable pre-training method to date.
Introduction
For years, the standard for pre-training large language models has been Masked Language Modeling (MLM), where models like BERT or RoBERTa predict missing words in a sentence. While effective, MLM often encourages models to memorize local facts rather than understanding high-level semantic relationships. Furthermore, achieving strong cross-lingual performance typically requires massive amounts of parallel text (bitext), which is scarce for many languages.
MARGE (Multilingual Autoencoder that Retrieves and Generates) addresses these limitations by treating paraphrasing and translation as primary objectives. Instead of filling in blanks, the model learns to reconstruct a target document by retrieving and synthesizing related information from a massive multi-lingual corpus.
Methodology
The core of MARGE is a retrieval-augmented sequence-to-sequence framework. The model operates through two main components that are trained jointly from a random initialization:
- Retrieval Model: This component computes relevance scores between a target document and potential evidence documents using cosine similarity between embeddings.
- Reconstruction Model: A standard Transformer seq2seq model where the encoder processes retrieved evidence and the decoder attempts to generate the original target document.
Crucially, the retrieval scores bias the decoder's cross-attention mechanism. If the decoder successfully reconstructs the target using a specific evidence document, the gradient "rewards" the retrieval model. This allows the model to bootstrap from zero, initially finding documents with lexical overlap and gradually developing the ability to identify semantically related documents across different languages.
Key Results
MARGE was pre-trained on the CC-NEWS and Wikipedia corpora across 26 languages. The model demonstrates remarkable capabilities in both zero-shot and supervised settings.
Zero-Shot Translation and Summarization
Without ever seeing a single pair of translated sentences during training, MARGE achieved a 35.8 BLEU score on German-to-English translation. It also outperformed extractive mBERT baselines on the MLSum dataset, showing it can effectively synthesize information across multiple documents.
Superior Retrieval Performance
Because retrieval is central to its pre-training, MARGE becomes an exceptionally strong retriever. On the BUCC sentence retrieval task, MARGE outperformed other unsupervised models like mBERT and XLM by nearly 10 points.
Discussion and Conclusion
MARGE demonstrates that paraphrasing can serve as a "super-task" that naturally captures translation, summarization, and information retrieval. By replacing the local noise of MLM with the semantic noise of related documents, the model develops deeper, more robust cross-lingual representations. While challenges remain regarding the computational overhead of rebuilding document indices and performance consistency across all language families, MARGE represents a significant step toward more generally applicable pre-training methods.
Summary updated