Multilingualism Encourages Recursion: a Transfer Study with mBERT

Andrea De Varda, Roberto ZamparelliNAACL 2022

TL;DR

This paper investigates how multilingual pre-training in models like mBERT influences their ability to understand hierarchical linguistic structures, compared to their monolingual counterparts (BERT). They found that multilingual models show a stronger reliance on hierarchical cues, suggesting that exposure to diverse languages encourages deeper structural understanding.

What Problem Does This Paper Address?

Massively Multilingual Models (MMMs) like mBERT are powerful tools that can perform NLP tasks across many languages with a single model. While their practical benefits are clear (reducing resource needs and improving performance on low-resource languages), their internal workings and the linguistic knowledge they acquire are still areas of active research. Specifically, it's unclear whether multilingual training leads to a fundamentally different understanding of language structure compared to monolingual training. This paper aims to shed light on this by testing how well mBERT and BERT can generalize their learned patterns to artificial languages with varying degrees of structural complexity, focusing on hierarchical relationships.

Methods

The researchers employed a novel evaluation procedure called the iterative token-level cloze task (ITCT). This is an adaptation of BERT's native masked language modeling (MLM) task. Instead of masking a small portion of tokens, ITCT masks all tokens iteratively. For each token, the model predicts it based on the surrounding context. The average probability assigned to the masked tokens across all timesteps serves as an index of the sequence's predictability.

They froze the weights of both BERT (monolingual) and mBERT (multilingual) and tested their ability to predict tokens in artificial corpora. This "zero-shot transfer" approach allows them to assess the models' inherent structural understanding without further training. They specifically designed artificial corpora to isolate the effects of token distribution (Zipf's law) and hierarchical structure (nested vs. flat dependencies).

Datasets

The study used four custom-generated artificial corpora:

  • Random corpus: Tokens sampled uniformly, serving as a baseline.
  • Zipf's corpus: Tokens sampled according to a Zipfian distribution, mimicking natural language frequency patterns.
  • Flat brackets corpus: Sequences with paired tokens where dependencies do not necessarily nest and can cross.
  • Nested brackets corpus: Sequences with paired tokens where dependencies are strictly nested hierarchically.

These corpora were designed to have identical vocabulary sizes and sequence lengths to ensure fair comparisons, isolating the impact of structural properties.

Key Insights and Ideas

  • Multilingualism fosters hierarchical understanding: The core finding is that mBERT exhibits a stronger preference for and reliance on hierarchical structures compared to BERT. This suggests that exposure to multiple languages during pre-training encourages models to learn more abstract and generalized structural patterns.
  • Zipf's Law is universally beneficial: Both BERT and mBERT performed better on sequences with a Zipfian token distribution compared to random distributions, indicating that both models learn to expect non-uniform token frequencies, consistent with natural language.
  • Pairing matters, nesting is a multilingual advantage: Both models benefited from paired token dependencies (flat brackets) over simple Zipfian distributions. However, only mBERT showed a significant advantage when moving from flat to nested structures, highlighting the multilingual model's enhanced ability to capture recursion.
  • Monolingual BERT's reliance on local heuristics: The study suggests that monolingual BERT might be more influenced by local, linear ordering cues and the repetition of identical adjacent tokens, potentially masking its understanding of deeper hierarchical structures.

Key Results

  • Both BERT and mBERT showed significantly higher performance on the Zipf's corpus compared to the Random corpus.
  • Both models showed a substantial performance increase when transitioning from the Zipf's corpus to the Flat brackets corpus, indicating the importance of paired token dependencies.
  • A key dissociation emerged when comparing the Flat brackets and Nested brackets corpora:
  • mBERT showed a significant, albeit small, performance advantage in the Nested brackets corpus, indicating a preference for hierarchical structures.
  • BERT showed the opposite trend, performing better on the Flat brackets corpus, suggesting it is less sensitive to or even hindered by nested recursion.
  • Statistical tests confirmed these differences, with the contrast between flat and nested structures showing significant but opposite effects for BERT and mBERT.

Limitations and Future Work

The authors acknowledge that their study used artificial languages to maximize experimental control. They suggest that future research should evaluate these findings in more naturalistic settings to assess the generalizability of the observed structural biases in multilingual models. They also note that while mBERT shows a hierarchical bias, it doesn't imply BERT cannot capture paired relationships; rather, it suggests mBERT is better equipped to exploit them when they become more complex.

Summary updated

BibTeX
@inproceedings{varda2022multilingualism,
  author = {Andrea De Varda and Roberto Zamparelli},
  title = {{Multilingualism Encourages Recursion: a Transfer Study with mBERT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure