Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
Lists
Only you see this
Abstract
Reliable multilingual evaluation is difficult, and culturally appropriate evaluation is even harder to achieve.A common practice to fill this gap is to machine-translate English evaluation sets. However, translation introduces language bias and carries over cultural and regional assumptions from the original questions – often testing knowledge irrelevant to the target audience. In this work, we highlight the extent and impact of these biases and present a multilingual evaluation framework that aims to mitigate them through improved translations and annotation practices.Through a large-scale study involving professional and community translators and annotators, we show that state-of-the-art models excel primarily by learning Western-centric concepts. Notably, we find that model rankings on the full MMLU change when evaluated on a subset of questions explicitly marked as culturally sensitive.We release Global MMLU, a multilingual extension of MMLU across 42 languages, featuring improved translation quality, expanded language coverage, and designated subsets labeled as culturally sensitive and culturally agnostic to enable a more comprehensive and equitable benchmark for evaluating language models across diverse linguistic and cultural contexts.
BibTeX
@inproceedings{singh2025global,
author = {Shivalika Singh and Angelika Romanou and Clémentine Fourrier and David Ifeoluwa Adelani and Jian Gang Ngui and Daniel Vila-Suero and Peerat Limkonchotiwat and Kelly Marchisio and Wei Qi Leong and Yosephine Susanto and Raymond Ng and Shayne Longpre and Sebastian Ruder and Wei-Yin Ko and Antoine Bosselut and Alice Oh and André F. T. Martins and Leshem Choshen and Daphne Ippolito and Enzo Ferrante and Marzieh Fadaee and Beyza Ermis and Sara Hooker},
title = {{Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation}},
booktitle = {Annual Meeting of the Association for Computational Linguistics},
year = {2025},
}