Abstract
The Massive Multitask Language Understanding (MMLU) benchmark serves as a primary metric for Large Language Model (LLM) performance. However, its reliance on Western-centric content and machine-translated versions for non-English evaluation introduces significant biases. We present Global-MMLU, a comprehensive dataset covering 42 languages that categorizes questions by cultural sensitivity. Our analysis reveals that 86.5% of culturally sensitive questions are Western-specific, and model rankings shift by an average of 5.7 positions when Western "home-field advantage" is removed. These findings underscore the necessity of culturally aware and human-verified benchmarks for truly global AI evaluation.
Introduction
As Large Language Models (LLMs) are deployed globally, the standard for measuring their "intelligence"—the Massive Multitask Language Understanding (MMLU) dataset—faces scrutiny. While MMLU is effective for English-centric evaluation, multilingual versions are typically created through simple machine translation. This approach suffers from two critical flaws: Cultural Bias, where questions are deeply rooted in Western contexts (e.g., US Law), and Translation Artifacts, where "translationese" phrasing can distort a model's true reasoning capabilities.
Methodology: Building Global-MMLU
To address these issues, we developed a multi-pronged pipeline to "de-bias" and "de-noise" multilingual evaluation. This involved a massive human annotation effort and a refined translation strategy.
Human Annotation and Categorization
We collaborated with 200 professional and community annotators to review 2,850 MMLU questions. Each question was tagged based on the type of knowledge required:
- Culturally-Sensitive (CS): Questions requiring specific cultural, regional, or dialectal knowledge.
- Culturally-Agnostic (CA): Questions based on universal principles, such as mathematics or physics.
High-Quality Translation
Moving beyond raw machine translation, we employed professional and community post-editing for 42 languages. This ensures that the evaluation reflects natural, fluent language rather than the awkward phrasing often found in machine-translated web data.
Categorizing Cultural Knowledge
The annotation process identified three primary sub-categories of cultural sensitivity that often go overlooked in standard benchmarks.
- Cultural Knowledge: Information specific to particular social groups or traditions.
- Regional Knowledge: Facts tied to specific geographic locations (e.g., US voting behavior).
- Dialect Knowledge: Linguistic nuances and idioms specific to certain regions or communities.
Key Insights
The Western "Home-Field" Advantage
Our analysis confirms a heavy Western bias in the original MMLU. An astounding 84.9% of geographic questions focus on North America or Europe. Furthermore, of the 28% of questions identified as culturally sensitive, 86.5% are specific to Western culture. This suggests that high scores on standard MMLU may reflect a model's "Western-ness" rather than its general reasoning ability.
Ranking Volatility
A model’s rank is not a fixed property. When evaluating models on Culturally-Agnostic versus Culturally-Sensitive subsets, rankings shifted by an average of 5.7 positions. This volatility indicates that some models excel at "Western trivia" while others may possess stronger underlying reasoning capabilities that are obscured by cultural bias.
The Low-Resource Paradox
In low-resource languages, we observed that models sometimes perform better on machine-translated data than on human-verified data. This "low-resource paradox" likely stems from models being trained on low-quality, machine-translated web crawls. They become "fluent" in bad translations, but their performance drops when faced with natural, human-level language.
Conclusion
Global-MMLU provides a more rigorous and fair framework for evaluating AI on a global scale. By distinguishing between cultural knowledge and general reasoning, and by prioritizing human-verified translations, we can better understand the true capabilities and limitations of LLMs across different linguistic and cultural contexts. Future work will focus on expanding language coverage and increasing the granularity of regional tagging to further capture the world's diversity.
Summary updated