Abstract
XLM-RoBERTa (XLM-R) is a massive multilingual model trained on 2.5TB of filtered CommonCrawl data across 100 languages. By scaling up both data and model capacity, the authors significantly outperform previous benchmarks like mBERT and prove that a single multilingual model can reach state-of-the-art performance without sacrificing monolingual accuracy.
Introduction: The Curse of Multilinguality
In the quest for Cross-lingual Language Understanding (XLU), researchers face a fundamental trade-off known as the "curse of multilinguality." When a model is trained on many languages, it benefits from "positive transfer"—knowledge from high-resource languages like English helps the model understand low-resource languages like Swahili. However, if the model's capacity (number of parameters) remains fixed, adding more languages eventually dilutes the model's ability to represent any single language well.
Previous models like mBERT and the original XLM were limited by their training data (mostly Wikipedia, which is sparse for many languages) and their relatively modest sizes. XLM-RoBERTa investigates whether scaling everything up—data, vocabulary, and parameters—can break this bottleneck.
Methodology
The authors introduce XLM-RoBERTa (XLM-R), a Transformer-based masked language model. Key technical choices include:
- Massive Scaling: They trained on 2.5TB of data—two orders of magnitude more than previous efforts for low-resource languages.
- Masked Language Modeling (MLM): Unlike some previous XLM versions, they used only the unsupervised MLM objective, proving that supervised "Translation Language Modeling" isn't strictly necessary if you have enough data.
- SentencePiece Tokenization: They applied SentencePiece directly to raw text for all 100 languages, removing the need for language-specific pre-processing.
- Sampling Strategy: They used an exponential sampling rate ($\alpha = 0.3$) to balance batches, ensuring the model saw enough low-resource language data without being overwhelmed by English.
Datasets and Scaling
The primary training corpus, CC-100, is a newly curated version of CommonCrawl covering 100 languages. For many low-resource languages, this increased the available training data by a factor of 100 compared to Wikipedia.
Evaluation was conducted on benchmarks including XNLI (cross-lingual inference), MLQA (question answering), and GLUE (English-only tasks) to ensure the model remained competitive on monolingual benchmarks.
Analysis: Alleviating the Curse
A central finding of the paper is that the "curse of multilinguality" is alleviatable. While performance typically drops as more languages are added to a fixed-capacity model, increasing the model's capacity allows it to handle more languages without losing its edge.
Furthermore, the authors discovered that vocabulary capacity matters significantly. Increasing the vocabulary size (from 32k to 250k tokens) provided massive gains, suggesting that allocating more parameters to the embedding layer is often more beneficial than simply adding more layers to the Transformer itself.
Key Results
- XNLI: XLM-R achieved an average accuracy of 80.9%, outperforming mBERT by 14.6%.
- Low-Resource Gains: The improvements were most dramatic for languages like Swahili (+15.7% accuracy) and Urdu (+11.4%) compared to previous models.
- Monolingual Parity: On the GLUE benchmark (English), XLM-R (91.8 avg) stayed within 1% of the specialized RoBERTa model (92.8 avg), proving that multilinguality doesn't have to come at a "tax" to English performance.
Conclusion
XLM-R is the first model to show that a single "universal" model can perform as well as specialized monolingual models while being fluent in 99 other languages. While the computational cost is high—requiring 500 NVIDIA V100 GPUs—it demonstrates that scaling capacity and data is a viable path toward truly global language understanding.
Summary updated