The Multilingual Evaluation Gap

213 concepts185 papers12 proposals

Modern natural language processing operates under a striking paradox. On one hand, we can easily train massively multilingual models that span hundreds of languages. By scraping billions of tokens of raw web text, models can be exposed to everything from high-resource languages like English to low-resource regional dialects.

On the other hand, our ability to verify whether these models actually understand what they are processing is severely bottlenecked. While pretraining data can be gathered automatically at scale, high-quality evaluation requires human expertise, cultural nuance, and rigorous gold-standard annotation. This asymmetry creates the multilingual evaluation gap: a massive blind spot where a model's downstream performance, safety, and reasoning capabilities remain completely unverified across the vast majority of the world's living languages.

The Asymmetry of Scale

To understand why this gap exists, we have to look at how data is sourced for pretraining versus evaluation. Pretraining relies on raw, unannotated text. For example, Christopher A. Choquette-Choo et al. introduced the MADLAD-400 dataset, which covers 419 languages and 2.8 trillion tokens by leveraging document-level language identification.

However, as the authors note, a significant chasm remains between the number of languages represented in these massive datasets and the availability of high-quality benchmarks to evaluate them. While we can train a model on hundreds of languages simultaneously, standard evaluation suites are overwhelmingly English-centric. As highlighted by Alexis Palmer et al. in the Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), this lack of standardized benchmarks across diverse languages and specialized domains severely hinders our ability to build systems that generalize effectively in a global context.

[Pretraining Scale]  =========================================> 400+ Languages (Raw Text)
[Evaluation Scale]   ===> 10-20 Languages (Gold-Standard Benchmarks)

Why Translation is an Imperfect Proxy

A common shortcut to bypass this benchmark scarcity is to take an existing English evaluation suite (like MMLU or GSM8K) and machine-translate it into target languages. While cost-effective, this approach introduces severe confounding variables:

  1. Translation Artifacts: Machine translation frequently introduces grammatical errors, awkward phrasing, or literal translations of idioms that alter the underlying difficulty of a question. In their work on the XQA dataset, Jiahua Liu et al. demonstrated that a significant performance gap remains between English and other languages due directly to translation errors and shifting question complexities.
  2. The Fine-Tuning Paradox: Relying on translated data can actively harm model performance. In an evaluation of instruction-tuned models adapted to Italian, Andre Freitas discovered that Italian-specific adapters often performed significantly worse than their original English counterparts. This suggests that low-quality, translated instruction data can degrade a model's core reasoning abilities rather than improving its linguistic adaptability.
  3. Cultural and Semantic Mismatch: Standard benchmarks fail to capture localized nuances. Anders Søgaard et al. showed in their study on the MozArt dataset that semantic and social biases vary wildly from one language to another within the same model, a phenomenon that small-scale or purely translated parallel benchmarks fail to capture.

Probing the Blind Spots Without Labeled Data

If human-annotated data is unavailable for thousands of low-resource languages, how can we know if a model understands them at all? Researchers have designed clever algorithmic workarounds to probe internal model representations without relying on traditional gold-standard benchmarks.

Monolingual Local Sensitivity

Instead of measuring task accuracy, we can measure how stable a model's internal representations are when subjected to structural perturbations. Louis Clouâtre et al. proposed a technique called Monolingual Local Sensitivity in their paper on Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes.

The intuition is elegant: if a model truly understands the syntax and structure of a language, scrambling the character or word order of an input sentence should drastically alter its internal hidden states. Conversely, if the language is completely unintelligible to the model (acting as mere noise), scrambling the characters will result in very little change to the model's representations. By measuring this delta using only unlabeled text, researchers can map out exactly which languages are blind spots for a model.

Zero-Shot Performance Prediction

Another approach is to treat performance evaluation as a meta-learning problem. When downstream evaluation data is missing for a target language, can we predict how well a model will perform based on its performance in other tasks and languages?

Kabir Ahuja et al. addressed this in Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models. By utilizing Multi-Task Learning (MTL), the authors demonstrated that knowledge can be shared across different NLP tasks to predict zero-shot performance in data-scarce scenarios, reducing prediction errors by up to 20% and offering a way to verify model safety before deployment.

Moving Beyond Saturation with Native Benchmarks

As multilingual models grow more powerful, existing cross-lingual benchmarks are hitting a saturation point. In XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation, Sebastian Ruder et al. pointed out that models are approaching human-level performance on standard, simple tasks like sentence retrieval. This saturation masks deep-seated deficiencies in complex reasoning and low-resource language support.

To combat this, a major coordinated effort is underway to build native, multi-task evaluation suites that reflect the unique linguistic and cultural landscapes of specific language families:

  • African Languages: Ife Adebara et al. introduced SERENGETI, a model covering 517 African languages, alongside the AfroNLU benchmark. This framework explicitly accounts for linguistic genealogy, helping close the gap between pretraining inclusion and downstream evaluation.
  • Nguni Languages: Francois Meyer et al. developed NGLUEni, the first comprehensive benchmark suite for the four Nguni languages of South South Africa, establishing a baseline to systematically track model performance in non-dominant contexts.
  • Norwegian: Addressing the narrow scope of mid-resource language evaluations, Vladislav Mikhailov et al. created NorEval, a human-curated benchmark spanning 24 datasets. Their evaluation revealed that even when models achieve low perplexity, they still lag significantly behind native speakers in complex reasoning and cultural nuance.

Ultimately, bridging the multilingual evaluation gap requires moving away from the assumption that a model's English capabilities serve as a universal proxy for intelligence. By developing native benchmarks and unsupervised structural probes, the NLP community is shifting toward an evaluation paradigm that is as globally distributed as the data powering the models themselves.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multilingual_evaluation_gap_28588383,
  title = {multilingual evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multilingual-evaluation-gap/rdr_28588383400c40528ff546634f8c848b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_multilingual_evaluation_gap_28588383_v1,
  title = {multilingual evaluation gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multilingual-evaluation-gap/rdr_28588383400c40528ff546634f8c848b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake