Abstract
In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.
Introduction
When Google released Multilingual BERT (M-BERT), it was trained on the monolingual Wikipedia pages of 104 languages simultaneously. Surprisingly, it demonstrated strong performance in cross-lingual transfer: a model fine-tuned for a task like Named Entity Recognition (NER) in English would "just work" for Spanish or even Japanese.
Crucially, M-BERT was not trained on parallel data (translation pairs). This raised a fundamental question: How is the model bridging the gap between languages? Is it simply memorizing shared vocabulary (e.g., "DNA", "Paris"), or is it learning a deeper, universal structure? This study probes the limits of M-BERT's multilinguality to understand what it captures and where it fails.
Methodology
The primary lens of investigation is zero-shot cross-lingual transfer. The authors fine-tune M-BERT on a specific task in one language and evaluate its performance on another. To dissect the causes of transfer success, several probing experiments were designed:
- Lexical Overlap Analysis: Quantifying the dependence of transfer on shared "word pieces" between languages.
- Cross-Script Transfer: Evaluating transfer between languages using entirely different scripts (e.g., Urdu in Arabic script to Hindi in Devanagari script) to isolate structural transfer from string matching.
- Typological Probing: Correlating transfer performance with grammatical similarities (e.g., word order) using the World Atlas of Language Structures (WALS).
- Feature Space Mapping: Analyzing internal activations to determine if sentences in different languages map to a shared multilingual representation.
The study utilizes benchmarks including CoNLL-2002/2003 (NER), Universal Dependencies (POS tagging), an in-house 16-language NER dataset, WMT16 parallel sentences, and a Hindi/English code-switching dataset.
Key Insights and Ideas
Beyond Vocabulary Memorization
A core finding is that M-BERT’s generalization capability is not solely dependent on shared vocabulary. While an English-only BERT (EN-BERT) fails as word overlap decreases, M-BERT maintains stable performance even with zero lexical overlap. This suggests it learns a structural mapping that is language-neutral.
The "Sweet Spot" of Multilinguality
Multilinguality is not uniform across the model's architecture. Probing internal activations reveals that the middle layers (approximately layers 6-8) are the most "multilingual." In contrast, lower layers are more sensitive to specific tokens, while higher layers are more specialized for the original language-modeling objective.
Structural and Typological Constraints
Transfer is significantly more effective between languages sharing typological features. For instance, similarity in word order (e.g., Subject-Verb-Object) facilitates knowledge transfer. Furthermore, M-BERT handles code-switched text surprisingly well, likely due to "anchor points" like numbers or URLs that force disparate languages into a shared space during pre-training.
Key Results
- Cross-Script Success: An M-BERT model fine-tuned only on Urdu (Arabic script) achieved 91% accuracy on Hindi (Devanagari script) POS tagging, despite the two languages sharing no common characters.
- M-BERT vs. EN-BERT: While an English-only BERT (EN-BERT) fails completely as word overlap decreases, M-BERT's performance remains stable across a wide range of lexical overlaps.
- Translation via Offset: The authors found they could "translate" sentences by simply calculating a mean vector offset between two languages. In middle layers, this method achieved over 50% nearest-neighbor translation accuracy for language pairs like English-German and English-Russian.
Discussion and Limitations
Despite its strengths, M-BERT faces two primary hurdles:
- Structural Divergence: The model struggles with "re-wiring" grammar. It can map words to known structures but fails to transform one grammatical structure (e.g., SVO) into another (e.g., SOV).
- Transliteration: Generalization to transliterated text (e.g., Hindi in Latin script) is poor if that specific combination was absent during pre-training.
Conclusion
M-BERT is a powerful tool for cross-lingual transfer, creating deep multilingual representations without parallel data. However, achieving true cross-lingual mastery will likely require explicit multilingual objectives to overcome deep structural barriers.
Summary updated