How Multilingual is Multilingual BERT?

Telmo Pires, Eva Schlinger, Dan Garrette

Abstract

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.

Introduction

When Google released Multilingual BERT (M-BERT), it presented a linguistic puzzle. Unlike previous cross-lingual models that relied on parallel corpora or explicit translation pairs, M-BERT was trained solely on the concatenated Wikipedia entries of 104 different languages. Despite this lack of cross-lingual supervision, the model demonstrated remarkable zero-shot cross-lingual transfer capabilities. A model fine-tuned for a task like Named Entity Recognition (NER) in English could be applied to Spanish or German without any additional training in those target languages. This study investigates the mechanisms behind this "accidental" multilingualism.

Probing the Mechanisms of Transfer

To understand M-BERT's performance, the researchers conducted extensive probing experiments using Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. The core methodology involved fine-tuning M-BERT on a task in one language and evaluating it on another. These results were compared against a monolingual English BERT (EN-BERT) baseline to isolate the effects of multilingual pre-training.

The Role of Vocabulary and Script

A primary hypothesis for M-BERT's success was the presence of "anchor points"—shared vocabulary or subword tokens across languages (e.g., "Paris", "Apple"). However, the authors found that M-BERT's transfer capability persists even when there is zero lexical overlap between the source and target languages.

Zero-shot F1 score vs. lexical overlap for M-BERT and English BERT.

As shown in Figure 1, while the performance of a monolingual model (EN-BERT) collapses as lexical overlap decreases, M-BERT maintains high F1 scores even at 0% overlap. This is most evident in transfer between languages with different scripts, such as Urdu (Arabic script) and Hindi (Devanagari script). Despite the visual difference in writing systems, M-BERT achieved 91% accuracy in transferring from Urdu to Hindi, suggesting it maps structural linguistic properties to a shared representation independent of the script.

Typological Constraints

While M-BERT is script-agnostic, it is heavily influenced by linguistic typology—the structural features of a language such as word order. Using the World Atlas of Language Structures (WALS), the researchers demonstrated that transfer performance is strongly correlated with the number of shared typological features.

Zero-shot accuracy as a function of common WALS features.

Figure 2 illustrates that as the number of common WALS features increases, the zero-shot accuracy improves. For instance, transferring between languages with the same Subject-Verb-Object (SVO) order (e.g., English to Spanish) is significantly more effective than transferring to a language with a different order, such as Subject-Object-Verb (SOV) (e.g., English to Japanese). This suggests that word order remains a significant bottleneck for the model's generalization.

Internal Representations and Alignment

By analyzing the model's hidden layers, the authors discovered that M-BERT naturally aligns different languages into a common subspace. The transformation required to map a sentence vector from one language to its translation in another is largely a constant offset. This implies that the model has learned a universal way to represent meaning that is largely independent of the specific language, though this alignment is most precise in the middle layers of the network.

Conclusion

M-BERT's ability to perform zero-shot transfer without parallel data is a result of its ability to learn deep, language-agnostic representations. However, these representations are not perfectly universal; they are constrained by the structural similarities between languages. Future work should focus on incorporating explicit multilingual objectives to bridge these typological gaps, particularly for language pairs with divergent word orders or those underrepresented in the pre-training data.

Summary updated

Abstract

In this paper, we show that Multilingual BERT (M-BERT), released by Devlin et al. (2018) as a single language model pre-trained from monolingual corpora in 104 languages, is surprisingly good at zero-shot cross-lingual model transfer, in which task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand why, we present a large number of probing experiments, showing that transfer is possible even to languages in different scripts, that transfer works best between typologically similar languages, that monolingual corpora can train models for code-switching, and that the model can find translation pairs. From these results, we can conclude that M-BERT does create multilingual representations, but that these representations exhibit systematic deficiencies affecting certain language pairs.
BibTeX
@inproceedings{pires2019how,
  author = {Telmo Pires and Eva Schlinger and Dan Garrette},
  title = {{How Multilingual is Multilingual BERT?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=xQtIAVwPoIC},
}
Figure