Reference: Weihao Xuan, Rui Yang, Heli Qi, et al. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
Abstract
MMLU-ProX is a high-difficulty multilingual benchmark covering 29 languages with 11,829 parallel questions per language. Designed to evaluate advanced reasoning rather than simple knowledge retrieval, MMLU-ProX addresses the "difficulty ceiling" of existing benchmarks. Through a sophisticated four-stage agentic translation pipeline and expert human verification, the study reveals that while top-tier models excel in Western European languages, a significant performance gap remains for low-resource languages, particularly those from Africa. The findings highlight the critical role of reasoning-enhanced architectures and model scaling in bridging the multilingual digital divide.
Introduction: The Dual Limitation of Multilingual Evaluation
As Large Language Models (LLMs) go global, evaluating them solely in English is no longer sufficient. However, current multilingual benchmarks face a "dual limitation":
- Difficulty Ceiling: Many existing benchmarks are based on the original MMLU, which modern models have largely "solved" or which focus more on factual knowledge than complex reasoning.
- Translation Quality vs. Parallelism: Human-translated benchmarks often lack parallel questions across all languages, while automated translations frequently introduce noise that masks a model's true reasoning capabilities.
MMLU-ProX bridges this gap by providing a challenging, high-quality, and strictly parallel dataset to evaluate if models can actually "think" in 29 different languages.
Methodology: Agentic Translation and Human Verification
To ensure high data quality, the researchers developed a four-stage agentic translation pipeline:
- Initial Translation: Claude 3.7 Sonnet generates the first draft.
- Self-Reflection: The model reviews its own work to refine terminology and remove superfluous explanations.
- Improvement: Meticulous editing based on the reflection stage.
- External Examination: A secondary "judge" model (OpenAI o3 or GPT-4.1) flags discrepancies for manual review.
This process was supplemented by over 400 hours of expert human labor. Professional native speakers verified 15 selected languages, scoring them on accuracy, fluency, and completeness to ensure the benchmark's integrity.
Key Insights and Results
The evaluation of 36 state-of-the-art models yielded several critical insights into the current state of multilingual AI.
The Reasoning Mode Advantage
Models with explicit "thinking" or reasoning-enhanced stages (like DeepSeek-R1 and Qwen3-Think) significantly outperform their standard counterparts. DeepSeek-R1 achieved the highest overall average score of 75.5% across all 29 languages. Interestingly, these models showed very little difference between zero-shot and 5-shot performance, suggesting their reasoning capabilities are internalized.
Asymmetric Scaling and the African Language Gap
Scaling model size provides "asymmetric benefits." While English performance improves steadily with size, low-resource languages like Wolof see massive jumps only after reaching a certain capacity (e.g., moving from 8B to 32B parameters), suggesting a "minimum capacity" is required for multilingual reasoning to emerge.
Despite these gains, a stark disparity remains. Top models hit ~80% accuracy in Western European languages but struggle to break 40-50% in African languages like Wolof and Yoruba, highlighting a persistent digital divide.
Conclusion
MMLU-ProX sets a new bar for multilingual evaluation by focusing on complex reasoning across a parallel dataset. While reasoning-enhanced models and scaling show promise, the persistent gap in low-resource language performance underscores the need for continued research into culturally localized data and inclusive AI development. The "Lite" version of the benchmark (658 questions) offers an efficient tool for developers to track these improvements.
Summary updated