Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features

Ester Hlavnova, Sebastian Ruder

Abstract

A challenge towards developing NLP systems for the world’s languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically-aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models’ behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.

Introduction

Modern NLP models are often evaluated on benchmarks that are heavily English-centric or provide only a high-level view of cross-lingual performance. This approach frequently masks a critical issue: languages differ significantly in their typological features—the structural rules governing word order, gender agreement, and complex verb conjugations. A model might achieve high accuracy on a general benchmark in a language like Slovak or Swahili, yet fail at practical tasks because it lacks a deep understanding of how that language handles negation, numerals, or spatial relationships.

To address this, we introduce M2C (Multilingual Morphological Checklist), a framework designed to "unit test" NLP models across 12 typologically diverse languages. M2C provides a fine-grained, morphologically aware testing environment that probes whether models truly grasp underlying linguistic structures beyond simple pattern matching.

The M2C Framework

The M2C framework represents an evolution of the original "CheckList" approach, specifically tailored for the complexities of non-English languages. Key innovations include:

  • Morphological Awareness: M2C integrates the UniMorph Schema, allowing users to define templates where placeholders (like nouns and adjectives) must maintain grammatical agreement (e.g., matching gender, number, or case).
  • Advanced Templating: The framework uses a specialized syntax to describe dependencies between words. For example, a template can specify that an adjective's form must depend on the gender of a specific name placeholder.
  • Answer Validation: Because a single question can have multiple grammatically correct answers in morphologically rich languages, M2C uses regex and morphological matching to validate model outputs rather than relying on simple string comparison.
The M2C framework overview, illustrating the pipeline from template creation to model evaluation.

Experimental Setup

We evaluated state-of-the-art models, including mT5-XXL and the PaLM family (S, M, L, and PaLM 2), across 12 languages: English, Spanish, Italian, French, German, Swedish, Finnish, Slovak, Russian, Swahili, Mandarin Chinese, and Arabic. The test suite covered two main categories:

  1. Language-agnostic features: Probing capabilities like negation, numerical reasoning (addition/subtraction), spatial expressions, temporal expressions, and comparatives.
  2. Language-specific features: Probing unique linguistic quirks such as the Swahili time system (based on sunrise/sunset), Finnish compounding possessives, Mandarin measure words, and Russian motion verbs.

Evaluation and Results

Our results indicate that while scaling models significantly improves general performance, it does not automatically solve deep-seated typological challenges. PaLM 2 emerged as the top performer, achieving an average accuracy of 92.8% across all tests, a substantial lead over mT5-XXL at 41.8%.

Model performance across languages and examples of morphological errors in Russian and Finnish.

Key Insights

  • The Morphology Bottleneck: A significant portion of model failures are "near misses." In these cases, the model understands the core concept but produces the incorrect word form—for instance, using the nominative case instead of the instrumental case in Slovak. In some tests, up to 50% of errors were purely morphological.
  • Persistent Blind Spots: Even the strongest models hit significant hurdles. For example, on language-specific tests, mT5-XXL scored 0% on Swahili time and Finnish possessives. While PaLM 2 improved this, it still only reached 69% on the Swahili time system.
  • Numerical Reasoning Hurdles: Accuracy drops sharply when models perform arithmetic in languages with complex numeral inflections, such as Russian or Slovak, despite performing well on identical math problems in English.
  • English Bias: Models consistently perform best in English. The performance gap between English and languages like Swahili or Arabic highlights a persistent English bias in the internal reasoning of "multilingual" models.

Conclusion and Future Work

M2C reveals that current state-of-the-art models still struggle with the structural nuances of diverse languages. While scaling helps, it is not a panacea for linguistic understanding. Future work should focus on "model-in-the-loop" template generation and expanding the framework to cover even more diverse languages and complex reasoning tasks beyond Question Answering. Addressing these morphological and typological blind spots is essential for creating truly global NLP systems.

Summary updated

Abstract

A challenge towards developing NLP systems for the world’s languages is understanding how they generalize to typological differences relevant for real-world applications. To this end, we propose M2C, a morphologically-aware framework for behavioral testing of NLP models. We use M2C to generate tests that probe models’ behavior in light of specific linguistic features in 12 typologically diverse languages. We evaluate state-of-the-art language models on the generated tests. While models excel at most tests in English, we highlight generalization failures to specific typological characteristics such as temporal expressions in Swahili and compounding possessives in Finish. Our findings motivate the development of models that address these blind spots.
BibTeX
@inproceedings{hlavnova2023empowering,
  author = {Ester Hlavnova and Sebastian Ruder},
  title = {{Empowering Cross-lingual Behavioral Testing of NLP Models with Typological Features}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=S4jQDntW87},
}
Figure