Evaluating morphological typology in zero-shot cross-lingual transfer

Antonio Martínez-García, Toni Badia, Jeremy BarnesACL 2021

TL;DR

This paper investigates how a language's morphological structure (how words are formed) impacts the performance of modern multilingual language models in zero-shot cross-lingual transfer tasks. They found that morphological typology significantly affects Part-of-Speech (POS) tagging, with fusional languages performing much better, while sentiment analysis is less sensitive.

What Problem Does This Paper Address?

Modern multilingual language models (like BERT and XLM-R) have revolutionized cross-lingual transfer, allowing models trained on one language to perform tasks in others without explicit parallel data. However, researchers have observed that this transfer isn't uniform; some languages consistently perform worse. This paper aims to systematically investigate why this happens, specifically focusing on the role of morphological typology – how languages structure words using morphemes (meaningful units). The authors hypothesize that languages with similar morphological structures will transfer better, and they want to quantify this effect and compare it to other factors like data size or domain mismatch.

Methods

The researchers used two state-of-the-art multilingual models: multilingual BERT (mBERT) and XLM-RoBERTa (XLM-R). They fine-tuned these models on two distinct NLP tasks:

  • Part-of-Speech (POS) Tagging: A structural task that identifies the grammatical role of words.
  • Sentiment Analysis: A semantic task that determines the emotional tone of text.

They experimented with 19 languages, carefully selected to represent four major morphological typologies: isolating, fusional, agglutinative, and introflexive. The core of their evaluation involved measuring zero-shot cross-lingual transfer performance, meaning models were trained on one language and tested on others without further training data for the target language. They analyzed performance using metrics like accuracy for POS tagging and macro F1 for sentiment analysis, and introduced the concept of transfer loss to quantify performance degradation when transferring between languages.

Datasets

The paper utilized existing datasets for their experiments:

  • Part-of-Speech Tagging: Data was primarily sourced from the Universal Dependencies project, which provides annotated POS tags for many languages. For Algerian, they used data from Seddah et al. (2020). Notably, they found no training sets for Thai and Cantonese, so these were used exclusively for testing.
  • Sentiment Analysis: No single centralized repository was used. Instead, data was collected from various sources and processed into binary (positive/negative) sentence-level sentiment datasets.

Key Insights and Ideas

  • Morphological Typology Matters, Especially for Structure: The paper strongly suggests that a language's morphological structure is a significant factor in cross-lingual transfer, particularly for tasks that rely on linguistic structure like POS tagging.
  • Fusional Languages Shine in POS Tagging: Models trained on fusional languages (like German) performed exceptionally well when transferring to other fusional languages for POS tagging. Conversely, introflexive languages consistently showed the lowest transfer performance.
  • Sentiment Analysis is More Robust: Sentiment analysis, being a more semantic task, appears less sensitive to morphological differences compared to POS tagging. While some differences exist, they are less pronounced.
  • Intra-group Transfer is Generally Better: Transferring to a language within the same morphological group generally results in less performance loss than transferring to a language from a different group.
  • XLM-R Outperforms mBERT in Transfer: Across both tasks, XLM-R generally demonstrated better cross-lingual transfer capabilities than mBERT, especially in sentiment analysis.
  • Language Specialization is a Concern: For XLM-R, fine-tuning on very large datasets for a specific language (like German for POS tagging) can lead to "language specialization," where the model becomes too tailored to that language, potentially hindering transfer to others.

Key Results

  • POS Tagging: Fusional models achieved over 80% accuracy when transferring within the fusional group. Introflexive languages consistently showed lower transfer performance, below 50%.
  • Sentiment Analysis: While less sensitive to typology, XLM-R models generally achieved higher macro F1 scores than mBERT.
  • Transfer Loss: The difference in transfer loss between intra-group and inter-group transfer was more pronounced in POS tagging than in sentiment analysis.
  • Regression Analysis: For POS tagging, the presence of the target language in pretraining had the strongest negative impact on transfer loss (-25.9 points), followed by transfer type (intra/inter-group) at -8.6 points.

Limitations and Future Work

  • Limited Scope of Typologies: While four major typologies were covered, the authors acknowledge that linguistic typology is complex and these categories are simplifications.
  • Script Ignored: The study did not investigate the effect of the script used by languages on cross-lingual transfer, which is difficult to control due to unique scripts and limited data.
  • Future Directions: The authors suggest exploring other tasks like dependency parsing or semantic role labeling, investigating specific language families for maximal benefit from shared morphology, and potentially training multilingual models on specific language families. They also note the potential for language specialization with large training sets and suggest further research into this phenomenon.

Summary updated

BibTeX
@inproceedings{martinezgarcia2021evaluating,
  author = {Antonio Martínez-García and Toni Badia and Jeremy Barnes},
  title = {{Evaluating morphological typology in zero-shot cross-lingual transfer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure