TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages

Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, Duygu AtamanACL 2025

Introduction

Evaluating the multilingual capabilities of Large Language Models (LLMs) has historically relied on benchmarks like MMLU, which are often machine-translated from English. This "translationese" approach frequently introduces errors, strips away cultural context, and fails to account for the complex morphosyntactic structures of non-Western language families. The Turkic language family, comprising over 180 million speakers, is particularly underserved by such methods. Existing benchmarks are typically limited to Turkish or suffer from translation artifacts.

To address this, we introduce TUMLU (Turkic Unified Multilingual Language Understanding), the first comprehensive, natively developed benchmark for the Turkic language family. Spanning 38,139 questions across 8 languages and 11 academic subjects, TUMLU provides a rigorous evaluation of how LLMs handle the unique nuances of languages like Uyghur, Tatar, and Karakalpak.

Methodology and Dataset Overview

TUMLU was constructed using a "native-first" approach, sourcing questions directly from official university entrance exams and middle/high school textbooks in their respective countries. This ensures that the benchmark reflects local educational standards and cultural knowledge rather than translated Western concepts.

Distribution of TUMLU dataset across languages and subject categories.

The dataset covers 11 categories, grouped into Humanities, Quantitative Sciences, Natural Sciences, and Social Sciences. To test script robustness, parallel datasets were created for languages like Uzbek and Kazakh, which are written in multiple scripts (Latin, Cyrillic, and Arabic). Models were evaluated in 5-shot and 5-shot Chain-of-Thought (CoT) settings to assess both raw knowledge and reasoning capabilities.

Key Insights: The Script Gap and Native Nuance

A significant finding from the TUMLU evaluation is the "script gap." Model performance is heavily influenced by the writing system used. For instance, Kazakh models performed better in Cyrillic, likely due to its dominance in web-crawled training data, while Uyghur performance peaked in the Arabic script.

Example of native questions in Uzbek (Latin and Cyrillic scripts) with English translations.

Furthermore, the benchmark highlights the importance of "native nuance." Models that excel at "universal" subjects like Physics often struggle with local history or literature, proving that high-quality, culturally grounded data is essential for true linguistic understanding. In low-resource scenarios like Crimean Tatar, models sometimes produce "synthesis" outputs—a hybrid of the target language and a more dominant relative like Turkish—indicating a gap between knowledge retrieval and generative fluency.

Key Results

  • Proprietary Models: Claude 3.5 Sonnet led the field with a mean score of 78.9% (5-shot), followed by GPT-4o at 74.9%.
  • Open-Source Performance: Qwen2.5-72B-Instruct emerged as the strongest open-source model (59.9%), outperforming Llama 3.1 and Gemma 2 variants of similar scale.
  • Reasoning and Scaling: Chain-of-Thought (CoT) prompting provided a consistent performance boost. For example, Claude 3.5 Sonnet’s score improved to 84.0% with CoT. Additionally, larger models consistently outperformed smaller ones, with Llama 3.1-70B beating its 8B counterpart by 18 percentage points.
  • Subject Trends: Performance was generally higher in Natural Sciences (Biology, Physics) than in Humanities, likely due to the cross-lingual consistency of scientific facts.

Conclusion and Future Directions

While TUMLU represents a major step forward, challenges remain. There is a difficulty imbalance across languages due to varying source material levels (e.g., middle school vs. university entrance). Future work will aim to include missing languages like Turkmen and expand beyond multiple-choice questions to better evaluate the linguistic quality of generated explanations. By providing a native-first evaluation framework, TUMLU sets a new standard for assessing LLMs in the Turkic linguistic landscape.

Summary updated

BibTeX
@inproceedings{isbarov2025tumlu,
  author = {Jafar Isbarov and Arofat Akhundjanova and Mammad Hajili and Kavsar Huseynova and Dmitry Gaynullin and Anar Rzayev and Osman Tursun and Aizirek Turdubaeva and Ilshat Saetov and Rinat Kharisov and Saule Belginova and Ariana Kenbayeva and Amina Alisheva and Abdullatif Köksal and Samir Rustamov and Duygu Ataman},
  title = {{TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure