Word-order Typology in Multilingual BERT: A Case Study in Subordinate-Clause Detection

Dmitry Nikolaev, Sebastian PadoNAACL 2022

TL;DR

This paper investigates how well Multilingual BERT (mBERT) understands syntactic structures across languages, specifically focusing on distinguishing main and subordinate clauses. They found that mBERT's performance is heavily influenced by word order, mirroring linguistic typologies, and that while it excels in some cross-lingual transfer scenarios, it struggles with languages that deviate from its training data's typical word order.

What Problem Does This Paper Address?

Pre-trained language models like BERT have shown remarkable capabilities, but their understanding of abstract grammatical patterns, especially across different languages, remains an active area of research. While some studies suggest BERT can learn similar syntactic representations across languages, others highlight challenges with specific grammatical phenomena and a drop in performance for typologically distant languages. This paper aims to add to this understanding by using the task of subordinate-clause detection as a probe. This task is chosen because it highlights cross-linguistic variation in clause structure and is relevant for downstream natural language understanding tasks. The authors want to see if BERT can generalize its understanding of this syntactic distinction to languages it hasn't been explicitly trained on (zero-shot learning) and how word order plays a role in this generalization.

Methods

The study employs two main experimental setups:

  1. Single-Language Models: Pre-trained BERT models (either general multilingual BERT or language-specific BERT variants) are fine-tuned on a dataset for a single language and then evaluated on that same language's test set. This establishes a baseline performance for each language.
  2. Zero-Shot Setting: A Multilingual BERT (mBERT) model is fine-tuned on a set of "source" languages. This fine-tuned model is then evaluated on "target" languages it has not been trained on. This setup directly tests the model's ability to transfer knowledge across languages.

For both setups, the task is framed as a binary classification problem: identifying whether a given predicate heads a main clause or a subordinate clause. The Universal Dependencies (UD) framework is used to define these clause types, with subordinate clauses identified by specific UD labels (acl, ccomp, advcl, csubj, xcomp). The last hidden state of the first subword of each predicate is fed into a two-layer MLP for classification.

Datasets

The primary dataset used is the Parallel Universal Dependencies (PUD) collection, which contains translations of 1000 English sentences. However, PUD corpora are typically test sets and are too small for fine-tuning. Therefore, the authors supplement PUD with other corpora for training and fine-tuning, covering a wide range of typologically diverse languages. The paper lists specific corpora used for various languages in Appendix A.2, including:

  • English: EWT (Universal Dependencies English Web Treebank)
  • Russian: SynTagRus
  • German: HDT (Hamburg Dependency Treebank)
  • Mandarin Chinese: GSD (Traditional Chinese Universal Dependencies Treebank)
  • Arabic: PADT (Prague Arabic Dependency Treebank)
  • Japanese: GSD (Japanese)
  • Korean: GSD (Korean)
  • Turkish: KENET (Turkish-Kenet UD Treebank)
  • French: GSD (French)
  • Welsh: CCG (Syntactic Corpus of Welsh)
  • Spanish: GSD (Spanish)
  • Irish: IDT (Irish UD Treebank)
  • Hebrew: HTB (Universal Dependencies Corpus for Hebrew)
  • Basque: BDT (Basque UD treebank)
  • Armenian: ArmTDP (Universal Dependencies treebank for Eastern Armenian)
  • Hindi: HDTB (Hindi Universal Dependency Treebank)
  • Vietnamese: VTB (UD version of the VLSP constituency treebank)

Key Insights and Ideas

  • Word Order Dominance: The most striking finding is that mBERT's zero-shot performance is heavily dictated by word order. Languages with similar word orders (e.g., VSO languages performing well on other VSO languages) show better transfer, suggesting that word order provides a strong inductive bias for this task.
  • The "Long Tail" of Complexity: While subordinate-clause detection is generally easy for BERT (achieving >90% accuracy in many single-language settings), there's a "long tail" of harder cases that prevent efficient generalization, especially in cross-lingual settings.
  • European Languages as Approximators: European languages with large training corpora (like Russian and English) can act as "general approximators," performing well across many target languages. However, they struggle with strict SOV languages.
  • SOV Languages are a Challenge: Strict SOV languages (like Japanese and Korean) are particularly difficult for models trained on other word orders. Even models trained on other SOV languages don't always transfer perfectly.
  • Annotation Discrepancies Matter: Differences in how linguistic phenomena (like quotations) are annotated across corpora can lead to errors, sometimes accounting for a significant portion of mistakes.
  • Syntactic Ambiguity: Genuine linguistic ambiguity, as seen in Mandarin constructions, also poses a challenge for models.

Key Results

  • Single-Language Performance: Models fine-tuned and tested on the same language achieve high accuracy, often above 90%, for most languages.
  • Zero-Shot Performance:
    • Russian, with the largest training corpus, achieved the best overall performance as a source language.
  • Arabic (VSO) showed excellent performance on Irish (VSO), demonstrating the word-order effect.
  • European languages (like Russian, English, French, German) performed well on many target languages but struggled with strict SOV languages (Japanese, Korean).
  • Mandarin Chinese proved difficult to generalize to, with no source language achieving accuracy above 71-72%.
  • Korean and Japanese consistently showed poor performance as target languages when trained on non-SOV languages.
  • Qualitative Findings: Analysis of English-to-Mandarin errors revealed that the model struggles with Mandarin's predominantly right-headed clause structure, which differs from English's more varied left-headedness.

Limitations and Future Work

  • Limited VSO Data: The study could only test the VSO word-order effect with one pair of VSO languages (Arabic and Irish), limiting the generalizability of this specific finding.
  • Annotation Influence: The authors acknowledge that annotation discrepancies can influence results, suggesting a need for more consistent annotation practices.
  • Attention Patterns: While preliminary checks suggest attention to complementizers is important, a deeper analysis of attention patterns is beyond the scope of this paper.
  • Future Work: The authors suggest that for certain tasks, training on a single source language might not be sufficient, and more complex cross-lingual projection strategies might be needed. They also highlight that even seemingly simple syntactic tasks can be hard for models, indicating a fundamental limit to cross-lingual transfer. Further research could explore how to better handle SOV languages and mitigate the impact of word-order differences.

Summary updated

BibTeX
@inproceedings{nikolaev2022word,
  author = {Dmitry Nikolaev and Sebastian Pado},
  title = {{Word-order Typology in Multilingual BERT: A Case Study in Subordinate-Clause Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure