Identifying Elements Essential for BERT's Multilinguality

Philipp Dufter, Hinrich SchützeEMNLP 2020 · OpenReview

Abstract: Philipp Dufter and Hinrich Schütze (2020) investigate why Multilingual BERT (mBERT) succeeds in zero-shot cross-lingual transfer without parallel training data. By isolating architectural and linguistic variables through a "Fake-English" synthetic language setup, they demonstrate that multilinguality is driven by parameter pressure, shared structural anchors, and consistent word order.

Introduction

The release of Multilingual BERT (mBERT) introduced a powerful tool for zero-shot cross-lingual transfer. Despite being trained on 104 languages using only a standard Masked Language Modeling (MLM) objective, the model effectively aligns disparate languages into a shared representation space. This research seeks to identify the specific "ingredients"—architectural constraints and linguistic properties—that necessitate this alignment.

Methodology: Isolating Variables with Fake-English

To study multilinguality efficiently, the authors utilized "BERT-small," a miniaturized version of the architecture. A key innovation is the creation of Fake-English, a synthetic language where English token IDs are shifted by a constant. This results in a language with zero vocabulary overlap but identical syntax and statistics to English.

The authors then performed ablation studies, systematically modifying architectural features—such as language-specific position embeddings—and linguistic features, like inverting word order. They also introduced knn-replace, a masking strategy that replaces masked tokens with cross-lingual nearest neighbors to provide a weak supervision signal.

Key Insights: The Mechanics of Alignment

The study reveals several critical factors that force BERT to become multilingual:

  1. The "Parameter Pressure" Hypothesis: Counter-intuitively, overparameterization can hinder multilinguality. Smaller models, facing limited capacity, are pressured to find common structural patterns across languages to minimize loss.
  2. Structural Anchors: Shared special tokens (e.g., [CLS], [SEP]) and the random replacement of tokens during MLM act as "anchors," providing the initial bridges for cross-lingual alignment.
  3. Word Order: Alignment relies heavily on shared structural backbones. When word order is inverted in one language, multilinguality effectively vanishes.
Visualization of English and Fake-English embeddings. The model aligns the two languages into a shared space despite zero vocabulary overlap.

Experimental Results

The authors demonstrated that multilinguality is fragile when specific constraints are removed. By combining architectural changes—such as language-specific positions and shifted special tokens—with increased model size, the multilinguality score plummeted from 0.70 to 0.00.

Performance across different ablation settings. The chart highlights how overparameterization and inverted word order significantly degrade cross-lingual performance.

Linguistic similarity is equally vital; inverting word order reduced performance to near zero (0.01), even with all architectural anchors present. Conversely, the knn-replace strategy improved zero-shot accuracy on real-world tasks like XNLI by up to 9% for languages like Hindi.

Conclusion

This work clarifies that mBERT's multilinguality is not accidental but a result of specific architectural pressures and linguistic similarities. While the "small-scale" setup provides clear insights into necessary conditions, future research is needed to determine how these dynamics scale to massive models and more diverse language pairs.

Summary based on: Dufter, P., & Schütze, H. (2020). Identifying Elements Essential for BERT's Multilinguality. EMNLP.

Summary updated

BibTeX
@inproceedings{dufter2020identifying,
  author = {Philipp Dufter and Hinrich Schütze},
  title = {{Identifying Elements Essential for BERT's Multilinguality}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=dQ_Gc41gMTm},
}
Figure