Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes

Louis Clouâtre, Prasanna Parthasarathi, Amal Zouaq, Sarath Chandar

Abstract

Providing better language tools for low-resource and endangered languages is imperative for equitable growth. Recent progress with massively multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages. However, this transfer is not universal, with many languages not currently understood by multilingual approaches. It is estimated that only 72 languages possess a "small set of labeled datasets" on which we could test a model's performance, leaving the vast majority of languages without the resources available for simple evaluation. In this work, we attempt to clarify which languages do and do not currently benefit from such transfer. To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language. We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.

Introduction

Massively multilingual pretrained models, such as mBERT and XLM-R, have revolutionized Natural Language Processing (NLP) by enabling "zero-shot" transfer. This allows a model trained on a high-resource language like English to perform tasks in languages like Swahili or Quechua without additional training. However, this capability is far from universal. While there are over 7,000 living languages, current evaluation benchmarks cover fewer than 100. For the thousands of remaining low-resource languages, we lack the labeled datasets—such as parallel translations or sentiment labels—necessary to determine if a model actually understands them.

This paper addresses this "blind spot" by introducing a method to identify unintelligible languages using only raw, unlabelled text. By detecting where models fail before deployment, researchers can better prioritize data collection and model refinement for underserved communities.

Methodology: The Local Structure Probe

The authors propose a "Local Structure Probe" based on a fundamental hypothesis: if a model truly understands a language, its internal representations should be sensitive to the specific order of characters and words. Conversely, if a model's representation of a sentence remains unchanged when the sentence is scrambled, the model is likely not "reading" the linguistic structure at all.

Neighbor Flipping

To test this, the researchers employ a perturbation technique called "neighbor flipping." Adjacent characters in a sentence are swapped with a controllable probability ($\rho$). This allows the researchers to measure two things:

  1. Local Structure Degradation: Quantified using the chrF-2 metric, which measures character n-gram overlap between the original and scrambled text.
  2. Monolingual Local Sensitivity: This is the core metric. The researchers take 1,000 unlabelled sentences in a target language and create scrambled versions. They then measure how the model's ability to "match" the original sentence to its scrambled counterpart degrades as the level of scrambling increases.

If the model treats a coherent sentence and its scrambled version as nearly identical, it indicates a lack of structural understanding.

Experimental Setup

The study evaluates the sensitivity of three major models: mBERT, XLM-R, and Canine (a tokenization-free model). The evaluation is conducted across two main scales:

  • Validation Set: 7 standard NLP tasks (including XNLI and Tatoeba) across 44 languages to correlate sensitivity with known performance.
  • Large-Scale Probe: A massive sentence similarity task using the MTData dataset, covering 350 languages, many of which are extremely low-resource and previously unstudied in this context.

Key Insights

Sensitivity as a Necessary Condition

The researchers found a near-perfect correlation ($r=0.99$) between a model's reliance on local structure and its actual task performance. This suggests that local structure sensitivity is a universal requirement for model efficacy. If a model is insensitive to the arrangement of characters, it consistently fails at downstream tasks.

The 0.99 Threshold

The study identified a clear empirical boundary: any language with a monolingual local sensitivity score below 0.99 was virtually guaranteed to have poor representation within the model. This provides a simple, quantitative heuristic for researchers to identify "blind spots" without needing any labeled data.

Beyond Tokenization

Interestingly, the sensitivity patterns were consistent even for Canine, a model that operates directly on characters rather than a fixed vocabulary (tokens). This indicates that the failure to understand low-resource languages is not merely a "vocabulary problem" or a result of sub-optimal tokenization, but a deeper failure of the model to learn the underlying grammar and syntax during pretraining.

Geographic and Linguistic Impact

By mapping these sensitivity scores globally, the authors identified significant geographic clusters where current multilingual models are effectively "blind." These include:

  • Sub-Saharan Africa
  • Central America
  • The Pacific Islands

Specific examples include languages like Lao. Despite having 30 million speakers, Lao is poorly understood by current models, highlighting that a language's digital presence—rather than its speaker count—is the primary driver of model performance.

Conclusion and Future Work

While the Local Structure Probe is a powerful tool for identifying model failure, it has limitations. High sensitivity is a necessary condition for understanding, but not a sufficient one; a model might be sensitive to structure but still lack the semantic depth to solve complex tasks. Additionally, the method still requires approximately 1,000 written sentences, which may be unavailable for strictly oral languages.

However, as a "cheap" and scalable diagnostic, this approach allows the NLP community to move beyond guessing. By identifying exactly where models are blind, researchers can more effectively target data collection efforts to ensure that the benefits of AI are shared more equitably across the world's languages.

Summary updated

Abstract

Providing better language tools for low-resource and endangered languages is imperative for equitable growth. Recent progress with massively multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages. However, this transfer is not universal, with many languages not currently understood by multilingual approaches. It is estimated that only 72 languages possess a "small set of labeled datasets" on which we could test a model's performance, the vast majority of languages not having the resources available to simply evaluate performances on. In this work, we attempt to clarify which languages do and do not currently benefit from such transfer. To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language. We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.
BibTeX
@inproceedings{clouatre2022detecting,
  author = {Louis Clouâtre and Prasanna Parthasarathi and Amal Zouaq and Sarath Chandar},
  title = {{Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=2SP0pXNMfy},
}

Loading related content...

Figure