Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes

Louis Clouâtre, Prasanna Parthasarathi, Amal Zouaq, Sarath Chandar

Introduction

Massively multilingual models, such as mBERT and XLM-R, have demonstrated remarkable "zero-shot" capabilities, allowing them to perform tasks in languages they were never explicitly trained on. However, this success is often restricted to a small subset of the world's 7,000+ languages. Current evaluation benchmarks typically cover fewer than 50 languages, and experts estimate that only about 72 languages possess sufficient labeled data for reliable performance assessment. This leaves a significant "blind spot" for low-resource and endangered languages, where it is currently impossible to determine if a model is genuinely processing the language or merely producing noise.

The Local Structure Hypothesis

To address this gap, researchers propose a diagnostic tool that requires only raw, unlabeled text. The core hypothesis is straightforward: if a model truly "understands" a language, its internal representations must be sensitive to the local structure—specifically, the order of characters. If a model cannot distinguish between a coherent sentence and a scrambled version of the same text, it likely lacks a functional representation of that language.

Methodology: Neighbor Flipping and Local Sensitivity

The authors introduce a perturbation technique called neighbor flipping to test this sensitivity. In this process, adjacent characters are swapped with a specific probability $\rho$. For example, the sentence "The scholar is typesetting" might be transformed into "hT eshcoarl i stpyseteitn.g."

Illustration of the neighbor flipping perturbation technique.

By measuring the correlation between the degree of scrambling (quantified by character n-gram overlap, or chrF) and the resulting drop in model performance, the researchers define a metric called Local Sensitivity. To make this applicable to languages without labeled data, they developed the Monolingual Local Sensitivity probe. This probe uses a sentence retrieval task: the model must match 1,000 original sentences to their scrambled counterparts. A model that fails to see a significant difference between the two versions is flagged as having failed to grasp the language's local structure.

Empirical Validation

The researchers validated this approach across 44 languages using standard benchmarks like XNLI (inference) and Tatoeba (retrieval). The results revealed a near-perfect Pearson correlation ($r = 0.99$) between a model's sensitivity to local structure and its actual task performance.

Correlation between character-level perturbations and model performance.

This high correlation suggests that local structure sensitivity is a universal requirement for language understanding in these models. The study identified a critical "unintelligibility threshold": if a language's monolingual local sensitivity score falls below 0.99, the model almost certainly fails to represent that language accurately. In tasks like XNLI, no language below this threshold performed better than random chance.

Global Mapping and Insights

By applying this probe to 350 languages using the MTData collection, the authors created a global map of model performance. The findings highlight significant geographic disparities. While European and Southeast Asian languages are generally well-supported, models struggle immensely with languages from Sub-Saharan Africa, Central America, and the Pacific Islands.

Interestingly, the probe revealed that high speaker populations do not guarantee model proficiency. For instance, Lao, with 30 million speakers, was flagged as poorly understood, highlighting the need for better data representation beyond just population metrics. Furthermore, the reliance on local structure was consistent across different architectures, including subword-based models (mBERT, XLM-R) and character-based models (Canine), suggesting the issue lies in the learning process rather than just tokenization.

Limitations and Future Directions

While the Monolingual Local Sensitivity probe is a powerful diagnostic tool, it has limitations. It primarily offers high precision but lower recall; while low sensitivity always indicates poor performance, high sensitivity does not guarantee success, as a model might understand structure but lack semantic depth. Additionally, the method still requires written text, making it inapplicable to purely oral languages.

Despite these hurdles, the probe serves as a vital "canary in the coal mine," allowing researchers to identify unintelligible languages and prioritize them for future data collection and model refinement.

Summary updated

Abstract

Providing better language tools for low-resource and endangered languages is imperative for equitable growth. Recent progress with massively multilingual pretrained models has proven surprisingly effective at performing zero-shot transfer to a wide variety of languages. However, this transfer is not universal, with many languages not currently understood by multilingual approaches. It is estimated that only 72 languages possess a "small set of labeled datasets" on which we could test a model's performance, the vast majority of languages not having the resources available to simply evaluate performances on. In this work, we attempt to clarify which languages do and do not currently benefit from such transfer. To that end, we develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language. We construct a cross-lingual sentence similarity task to evaluate our approach empirically on 350, primarily low-resource, languages.
BibTeX
@inproceedings{clouatre2022detecting,
  author = {Louis Clouâtre and Prasanna Parthasarathi and Amal Zouaq and Sarath Chandar},
  title = {{Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=2SP0pXNMfy},
}
Figure