Morph Call: Probing Morphosyntactic Content of Multilingual Transformers

Vladislav Mikhailov, Oleg Serikov, Ekaterina ArtemovaNAACL 2021

TL;DR

This paper introduces "Morph Call," a new suite of 46 probing tasks designed to investigate the morphosyntactic knowledge embedded within multilingual transformer models across four Indo-European languages. By analyzing how these models handle guided sentence perturbations, the authors reveal surprising insights into how linguistic features are represented and how fine-tuning for tasks like POS-tagging can impact this knowledge.

What Problem Does This Paper Address?

Transformer-based language models have achieved remarkable success in Natural Language Processing (NLP), but understanding how they achieve this performance, particularly their grasp of linguistic phenomena beyond simple word meaning, remains a significant challenge. While much research has focused on English and higher-level linguistic features like syntax and semantics, the morphosyntactic properties of other languages—how words change form to convey grammatical information (like tense, number, or case)—have been largely under-explored, especially in multilingual models. This paper aims to fill this gap by systematically probing multilingual transformers for their understanding of these crucial grammatical features.

Methods

The core of this work is the Morph Call probing suite, which introduces a novel task type: guided sentence perturbations. Instead of relying solely on existing datasets or simple classification tasks, Morph Call generates perturbed sentences by systematically altering specific morphosyntactic features. This allows for a more controlled and nuanced evaluation of model knowledge.

The authors employ a combination of introspection techniques:

  • Probing Classifiers: Standard linear classifiers (Logistic Regression) are trained on model representations to predict specific linguistic properties.
  • Neuron Analysis: Investigating individual neurons to identify those most relevant for predicting a given property.
  • Layer-wise Analysis: Examining how performance changes across different layers of the transformer, revealing where knowledge is localized.
  • Correlation Analysis (ckasim): Measuring the similarity between representations of original and perturbed sentences to understand how well models recover from disruptions.

They also establish count-based and distributive baselines to contextualize the transformer models' performance.

Datasets

The study utilizes sentences from the Universal Dependencies (UD) corpus for four Indo-European languages: Russian, French, English, and German. These languages were chosen for their varying morphological richness. The UD corpus provides a consistent source for linguistic annotations across languages. The authors also mention using pymorphy2 for Russian, lemminflect for English, and Wiktionary for French and German to generate inflectional candidates for perturbations.

Key Insights and Ideas

  • Morphosyntactic Knowledge is Surprisingly Similar Across Languages: Despite differences in morphological complexity, multilingual transformers seem to learn and represent morphosyntactic properties in a remarkably consistent way across the tested languages.
  • Fine-tuning Can Be a Double-Edged Sword: While fine-tuning for tasks like POS-tagging can sometimes improve probing performance, it can also lead to significant drops or unstable behavior, particularly in tasks like "Masked Token," suggesting that fine-tuning might disrupt or partially erase certain learned knowledge.
  • Knowledge Localization Varies by Task and Language: Morphosyntactic features are often found in the middle-to-higher layers of transformers. However, this localization can shift depending on the specific task (e.g., syntactic vs. inflectional perturbations) and the language's morphological characteristics.
  • Syntactic Perturbations are Easier Than Inflectional Ones: Models generally perform better on tasks involving the removal of stop words or articles compared to tasks that require understanding inflectional changes.
  • Distilled Models Mimic Teachers, Mostly: Distilled models (like D-BERT and MiniLM) tend to mirror the behavior and knowledge representation of their larger, pre-trained counterparts, though some subtle differences emerge.

Key Results

  • The Morph Call suite consists of 46 probing tasks across the four languages.
  • Performance on Morphosyntactic Features tasks generally plateaus in the middle layers (layers 5-8 for 12-layer models).
  • The Masked Token task proved more challenging, with fine-tuned models sometimes showing performance drops of up to 10%. XLM-R demonstrated more stable behavior post-fine-tuning compared to other models.
  • For Perturbations tasks, syntactic perturbations (like Article Removal) were generally solved better than inflectional ones. XLM-R was noted as being particularly sensitive to perturbations.
  • Fine-tuning for POS-tagging showed varied effects: it could improve performance by 2-4% on Morphosyntactic Features tasks but negatively impact Masked Token tasks.
  • The number of neurons required to learn a property could increase or decrease after fine-tuning, with Russian Case and French Person showing increases, and English Number showing a decrease.

Limitations and Future Work

The authors acknowledge that the UD Treebanks can have inconsistencies between languages, which might affect dataset sizes and comparability. They also note that the fine-tuning process for POS-tagging can lead to unpredictable behavior, and the exact mechanisms behind these changes warrant further investigation.

Future work suggested includes:

  • Extending the analysis to a more diverse set of languages with different morphological types.
  • Investigating the interdependence of perturbations and how disrupting one aspect of a sentence can create ambiguity elsewhere.
  • Exploring the correlation between the number of neurons and probe performance after fine-tuning.
  • Conducting fine-tuning experiments specifically for morphosyntactic analysis using resources like UniMorph.
  • Testing the hypothesis on knowledge distribution across layers with a broader range of morphologically contrasting languages.

Summary updated

BibTeX
@inproceedings{mikhailov2021morph,
  author = {Vladislav Mikhailov and Oleg Serikov and Ekaterina Artemova},
  title = {{Morph Call: Probing Morphosyntactic Content of Multilingual Transformers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure