The Roles of English in Evaluating Multilingual Language Models

Wessel Poelman, Miryam de LhoneuxarXiv 2024 · arXiv

This position paper from researchers at KU Leuven challenges a common practice in multilingual AI evaluation: using English as a "crutch" to prompt models. The authors argue that mixing English instructions with target-language data creates an "imprecise" evaluation that prioritizes raw task performance over genuine language understanding.

TL;DR

Researchers often use English prompts to evaluate multilingual models because these models are primarily instruction-tuned in English. This paper argues that this "mixed-prompt" approach treats English as a programming interface rather than a natural language, leading to benchmarks that measure English instruction-following rather than true multilingual understanding.

What Problem Does This Paper Address?

In the current landscape of Large Language Models (LLMs), English is the undisputed heavyweight. Most models undergo extensive "instruction tuning" (learning to follow directions) primarily in English. Consequently, when researchers want to test how well a model understands a low-resource language like Turkish or Amharic, they often use a prompt like: "The topic of the news [Turkish Sentence] is..."

The authors identify a fundamental tension here. While using English instructions often boosts a model's scores, it creates a "mixed-prompt" environment that is linguistically unnatural. This raises a critical question: Are we measuring the model's grasp of the target language, or its ability to follow English instructions while ignoring the "noise" of another language?

Methods

As a position paper, the work provides a conceptual framework rather than a new algorithm. The authors propose a taxonomy to categorize how English is used in evaluations, distinguishing between its Role, its Format, and the ultimate Goal of the researcher.

Figure 1 – Schematic overview of the different roles of English in multilingual LM evaluation.

As shown in Figure 1, the authors contrast two main paths:

  1. English as an Interface: English acts like a programming language to "glue" target language passages together. The goal here is maximizing Task Performance.
  2. English as a Natural Language: English is treated as just one of many languages. Prompts are either monolingual (entirely in the target language) or naturally code-switched. The goal here is Language Understanding.

Datasets

The paper critiques evaluation setups used in several prominent multilingual benchmarks, including:

  • SIB-200: A news topic classification dataset covering over 200 languages.
  • IrokoBench (AfriMMLU): A benchmark for African languages that uses multiple-choice questions.
  • MaLa-500: A model adapted for 500+ languages, often evaluated using English-heavy prompts.
  • Hendy et al. (2023) Translation Prompts: Zero-shot translation setups where the instructions and language names are always in English.

Key Insights and Ideas

  • The "Programming Language" Fallacy: When English is used as an interface, it functions like a keyword in a programming language (e.g., a while loop). However, unlike code, LLMs are sensitive to the specific nuances and "leakage" of English as a natural language.
  • Confounding Factors: Mixed-prompts don't just test the task; they inadvertently test English instruction-following, script-switching (e.g., moving between Latin and Ge'ez scripts), and even the model's ability to ignore grammatical errors in the English prompt itself.
  • Usability vs. Understanding: If a model requires an English prompt to function in another language, it fails the "natural language interface" test. A user shouldn't have to know English to interact with a model in their native tongue.
  • Imprecise Benchmarking: The authors argue that "task performance" is often used as a proxy for "understanding," but the two are diverging. A model might get a high score on a Turkish task using an English prompt, but fail completely if asked the same question in Turkish.

Key Results

The paper highlights a consistent finding across recent literature: models perform significantly better when prompts are in English. For example, the authors cite Zhang et al. (2023), who concluded that models achieve higher performance when tasks are presented in English across various multilingual LMs.

However, the authors argue this result is "hardly surprising" and potentially misleading. It reflects the model's training bias rather than its linguistic capability. They advocate for a shift toward native, target-language prompting, even if it results in lower initial scores, as it provides a more honest assessment of Multilingual Natural Language Understanding (MLU).

Limitations and Future Work

The primary hurdle to the authors' recommendation is the scarcity of instruction-tuning data in non-English languages. Creating high-quality, native prompts for hundreds of languages is expensive and time-consuming.

The authors suggest that instead of relying on English "shortcuts" to boost scores, the research community should focus on:

  • Developing better datasets for multilingual instruction tuning.
  • Improving model architectures to handle low-resource languages without English intervention.
  • Adopting more rigorous evaluation standards that explicitly state whether English is being used as an interface or a natural language.

Ultimately, the paper serves as a call for "linguistic honesty" in AI research, urging developers to build models that truly speak the world's languages, rather than just translating English instructions under the hood.

Summary updated

Abstract

Multilingual natural language processing is getting increased attention, with numerous models, benchmarks, and methods being released for many languages. English is often used in multilingual evaluation to prompt language models (LMs), mainly to overcome the lack of instruction tuning data in other languages. In this position paper, we lay out two roles of English in multilingual LM evaluations: as an interface and as a natural language. We argue that these roles have different goals: task performance versus language understanding. This discrepancy is highlighted with examples from datasets and evaluation setups. Numerous works explicitly use English as an interface to boost task performance. We recommend to move away from this imprecise method and instead focus on furthering language understanding.
BibTeX
@misc{poelman2024roles,
  author = {Wessel Poelman and Miryam de Lhoneux},
  title = {{The Roles of English in Evaluating Multilingual Language Models}},
  year = {2024},
  eprint = {2412.08392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08392},
}
Figure