Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Wei-Yin Ko, Madeline Smith, Antoine Bosselut, Alice Oh, André F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, Sara Hooker

Abstract

The Massive Multitask Language Understanding (MMLU) benchmark serves as a primary metric for Large Language Model (LLM) performance. However, its reliance on Western-centric content and machine-translated versions for non-English evaluation introduces significant biases. We present Global-MMLU, a comprehensive dataset covering 42 languages that categorizes questions by cultural sensitivity. Our analysis reveals that 86.5% of culturally sensitive questions are Western-specific, and model rankings shift by an average of 5.7 positions when Western "home-field advantage" is removed. These findings underscore the necessity of culturally aware and human-verified benchmarks for truly global AI evaluation.

Introduction

As Large Language Models (LLMs) are deployed globally, the standard for measuring their "intelligence"—the Massive Multitask Language Understanding (MMLU) dataset—faces scrutiny. While MMLU is effective for English-centric evaluation, multilingual versions are typically created through simple machine translation. This approach suffers from two critical flaws: Cultural Bias, where questions are deeply rooted in Western contexts (e.g., US Law), and Translation Artifacts, where "translationese" phrasing can distort a model's true reasoning capabilities.

Methodology: Building Global-MMLU

To address these issues, we developed a multi-pronged pipeline to "de-bias" and "de-noise" multilingual evaluation. This involved a massive human annotation effort and a refined translation strategy.

Figure 1: The Global-MMLU pipeline, illustrating the transition from the original MMLU to an annotated and translated global benchmark.

Human Annotation and Categorization

We collaborated with 200 professional and community annotators to review 2,850 MMLU questions. Each question was tagged based on the type of knowledge required:

  • Culturally-Sensitive (CS): Questions requiring specific cultural, regional, or dialectal knowledge.
  • Culturally-Agnostic (CA): Questions based on universal principles, such as mathematics or physics.

High-Quality Translation

Moving beyond raw machine translation, we employed professional and community post-editing for 42 languages. This ensures that the evaluation reflects natural, fluent language rather than the awkward phrasing often found in machine-translated web data.

Categorizing Cultural Knowledge

The annotation process identified three primary sub-categories of cultural sensitivity that often go overlooked in standard benchmarks.

Figure 2: Examples of Cultural, Regional, and Dialect Knowledge questions identified during the annotation process.
  1. Cultural Knowledge: Information specific to particular social groups or traditions.
  2. Regional Knowledge: Facts tied to specific geographic locations (e.g., US voting behavior).
  3. Dialect Knowledge: Linguistic nuances and idioms specific to certain regions or communities.

Key Insights

The Western "Home-Field" Advantage

Our analysis confirms a heavy Western bias in the original MMLU. An astounding 84.9% of geographic questions focus on North America or Europe. Furthermore, of the 28% of questions identified as culturally sensitive, 86.5% are specific to Western culture. This suggests that high scores on standard MMLU may reflect a model's "Western-ness" rather than its general reasoning ability.

Ranking Volatility

A model’s rank is not a fixed property. When evaluating models on Culturally-Agnostic versus Culturally-Sensitive subsets, rankings shifted by an average of 5.7 positions. This volatility indicates that some models excel at "Western trivia" while others may possess stronger underlying reasoning capabilities that are obscured by cultural bias.

The Low-Resource Paradox

In low-resource languages, we observed that models sometimes perform better on machine-translated data than on human-verified data. This "low-resource paradox" likely stems from models being trained on low-quality, machine-translated web crawls. They become "fluent" in bad translations, but their performance drops when faced with natural, human-level language.

Conclusion

Global-MMLU provides a more rigorous and fair framework for evaluating AI on a global scale. By distinguishing between cultural knowledge and general reasoning, and by prioritizing human-verified translations, we can better understand the true capabilities and limitations of LLMs across different linguistic and cultural contexts. Future work will focus on expanding language coverage and increasing the granularity of regional tagging to further capture the world's diversity.

Summary updated

BibTeX
@inproceedings{singh2025global,
  author = {Shivalika Singh and Angelika Romanou and Clémentine Fourrier and David Ifeoluwa Adelani and Jian Gang Ngui and Daniel Vila-Suero and Peerat Limkonchotiwat and Kelly Marchisio and Wei Qi Leong and Yosephine Susanto and Raymond Ng and Shayne Longpre and Wei-Yin Ko and Madeline Smith and Antoine Bosselut and Alice Oh and André F. T. Martins and Leshem Choshen and Daphne Ippolito and Enzo Ferrante and Marzieh Fadaee and Beyza Ermis and Sara Hooker},
  title = {{Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure