Unsupervised Cross-lingual Representation Learning at Scale

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov

Abstract

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.

Introduction

Before the introduction of XLM-R (XLM-RoBERTa), researchers faced a significant trade-off in cross-lingual NLP. Monolingual models trained on specific languages were highly accurate, but multilingual models like mBERT often suffered from "capacity dilution"—where a fixed model capacity spread across many languages led to a drop in per-language performance. Furthermore, reliance on Wikipedia for training data severely limited performance on low-resource languages like Swahili or Urdu, where Wikipedia content is sparse.

XLM-R addresses these challenges by scaling both model capacity and training data, demonstrating that it is possible to build a single, massive model that achieves state-of-the-art performance across 100 languages without sacrificing accuracy on any individual language.

Methods

XLM-R builds upon the RoBERTa (Robustly Optimized BERT Approach) and XLM frameworks. The model utilizes Masked Language Modeling (MLM), learning to predict hidden words across all 100 languages. A key technical innovation is the use of a single SentencePiece tokenizer trained on all languages, which simplifies the pipeline and improves handling of code-switching. Notably, the authors found that removing language embeddings allowed the model to better learn universal semantic relationships. To accommodate 100 languages, the "Large" version of XLM-R scales to 550 million parameters with a 250,000-token vocabulary.

Datasets and Scaling

The primary training corpus, CC-100, consists of 2.5TB of cleaned text from CommonCrawl. This represents a massive increase in scale compared to Wikipedia, particularly for low-resource languages.

Comparison of dataset sizes between CommonCrawl and Wikipedia across 100 languages. CommonCrawl provides orders of magnitude more data for low-resource languages.

Evaluation was conducted on standard benchmarks: XNLI for cross-lingual inference, MLQA and NER for question answering and entity recognition, and GLUE to ensure English-only performance remained competitive.

Key Insights: The Curse of Multilinguality

The authors identify the "Curse of Multilinguality": as more languages are added to a model of fixed size, performance initially improves due to positive transfer between similar languages but eventually declines due to capacity dilution.

The Curse of Multilinguality: Accuracy across language groups as the number of languages increases, illustrating the trade-off between positive transfer and capacity dilution.

Scaling is the primary antidote to this curse. By increasing model size and vocabulary (from 50k to 250k tokens), the "peak" of the performance curve can be pushed further out, allowing the model to maintain high performance across a larger set of languages.

Results

XLM-R established new state-of-the-art results across multiple benchmarks:

  • XNLI: Achieved 80.9% average accuracy, a 14.6% improvement over mBERT.
  • Low-Resource Languages: Significant gains were seen in Swahili (+15.7%) and Urdu (+11.4%) compared to previous XLM models.
  • Monolingual Competitiveness: On the GLUE benchmark, XLM-R performed within 1% of the monolingual RoBERTa model, proving that a single multilingual model can match specialized monolingual systems.

Conclusion and Future Work

While XLM-R demonstrates the power of scale, it also highlights the high computational cost of such models, requiring 500 Nvidia V100 GPUs for training. Future research should focus on achieving similar gains with more modest computational budgets to democratize high-performance multilingual NLP. Additionally, addressing the vocabulary bottleneck remains an open area for optimization.

Summary updated

Abstract

This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6% average accuracy on XNLI, +13% average F1 score on MLQA, and +2.4% F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7% in XNLI accuracy for Swahili and 11.4% for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code, data and models publicly available.
BibTeX
@inproceedings{conneau2020unsupervised,
  author = {Alexis Conneau and Kartikay Khandelwal and Naman Goyal and Vishrav Chaudhary and Guillaume Wenzek and Francisco Guzmán and Edouard Grave and Myle Ott and Luke Zettlemoyer and Veselin Stoyanov},
  title = {{Unsupervised Cross-lingual Representation Learning at Scale}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=S7cSNJRBWv},
}
Figure