Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Shijie Wu, Mark DredzeEMNLP 2019 · DOI

TL;DR

This paper investigates the surprising cross-lingual capabilities of BERT, a powerful pre-trained language model. The authors demonstrate that BERT, when trained on a massive multilingual dataset without explicit cross-lingual alignment, achieves competitive zero-shot cross-lingual transfer performance across a variety of NLP tasks and languages, challenging previous assumptions about the need for explicit cross-lingual supervision.

What Problem Does This Paper Address?

The paper addresses the challenge of building effective NLP models for languages with limited labeled data, a common issue in low-resource scenarios. Traditionally, cross-lingual transfer learning relied on methods like aligning monolingual word embeddings using parallel data or bilingual dictionaries. While these methods showed promise, the authors questioned whether a single, multilingual model trained without explicit cross-lingual signals could achieve similar or better performance. Specifically, they investigate if BERT's multilingual variant (mBERT), pre-trained on 104 languages, could generalize to new languages and tasks without any task-specific cross-lingual training data (zero-shot transfer).

Methods

The core of the paper's methodology involves evaluating mBERT on five diverse NLP tasks: Natural Language Inference (NLI), Document Classification, Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, and Dependency Parsing. The authors employ a zero-shot transfer setting, where models are trained on English data and then directly applied to target languages. They explore different fine-tuning strategies, including fine-tuning all parameters and freezing lower layers of mBERT, to understand optimal usage. Additionally, they investigate the extent to which mBERT retains language-specific information and the role of subword sharing in cross-lingual transfer.

Datasets

The paper utilizes several standard datasets for evaluation:

  • MLDoc: A subset of the Reuters corpus for document classification across 8 languages.
  • XNLI: A natural language inference dataset covering 15 languages.
  • CoNLL 2002 and 2003 NER datasets: For Named Entity Recognition across 4 languages.
  • Chinese NER dataset: For Named Entity Recognition.
  • Universal Dependencies (UD) Treebanks (v1.4 and v2.2): For Part-of-Speech Tagging (15 languages) and Dependency Parsing (31 languages).
  • WiLI-2018: Used for language identification to assess language-specific information retention.

Key Insights and Ideas

  • Surprising Cross-Lingual Power: mBERT exhibits remarkably strong zero-shot cross-lingual transfer capabilities across a wide range of tasks and languages, often outperforming models that explicitly leverage cross-lingual signals.
  • No Explicit Alignment Needed: The success of mBERT suggests that joint pre-training on a massive multilingual corpus can implicitly learn a shared cross-lingual representation space, negating the need for explicit cross-lingual supervision in many cases.
  • Layer Freezing Benefits: Freezing the lower layers of mBERT during fine-tuning can lead to performance improvements, particularly for tasks that benefit from higher-level semantic abstractions.
  • Language-Specific Information Retention: Despite its cross-lingual abilities, mBERT layers retain language-specific information, which is crucial for its ability to perform the cloze task during pre-training.
  • Subword Sharing Matters: The sharing of subwords across languages, facilitated by BERT's WordPiece tokenizer, plays a significant role in enabling effective cross-lingual transfer. A correlation is observed between the percentage of shared subwords and transfer performance.

Key Results

  • Competitive Performance: mBERT achieved competitive or even state-of-the-art performance on all five evaluated NLP tasks in a zero-shot cross-lingual setting.
  • Outperforming Embeddings: In four out of five tasks, mBERT outperformed traditional cross-lingual word embeddings.
  • Layer Freezing Gains: Freezing the bottom layers of mBERT generally improved performance, with optimal freezing points varying by task (e.g., n=6 for sentence-level tasks, n=3 for word-level tasks).
  • Language Identification Accuracy: All tested layers of mBERT achieved around 96% accuracy on language identification, indicating retention of language-specific details.
  • Subword Correlation: A strong positive correlation was observed between the percentage of shared subwords (type-level) and cross-lingual zero-shot transfer performance for most tasks.

Limitations and Future Work

  • Low-Resource Languages: While mBERT performs well, the paper notes that explicit cross-lingual supervision (like bitext) can lead to even stronger performance, especially in low-resource settings. Future work could investigate how to leverage weak supervision to further improve mBERT for these languages.
  • Linguistic Perspective: The authors suggest that a deeper linguistic analysis of mBERT's representations would be beneficial to understand why it generalizes so well across languages.
  • Task-Specific Layers: The paper uses standard task-specific layers on top of mBERT. Future work could explore more sophisticated cross-lingual task-specific architectures.
  • Subword Sharing Nuances: While subword sharing is beneficial, the paper hints at potential overfitting issues. Further research could explore strategies to mitigate this.

Summary updated

BibTeX
@inproceedings{wu2019beto,
  author = {Shijie Wu and Mark Dredze},
  title = {{Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1077},
}
Figure