SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala

Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, Taro Watanabe

TL;DR

SinhalaMMLU is the first comprehensive, culturally grounded multi-task benchmark designed specifically for Sinhala, an under-resourced Indo-Aryan language spoken by over 17 million people. Comprising 7,044 native examination questions across 30 subjects, the benchmark reveals that even state-of-the-art LLMs struggle heavily with non-Western cultural knowledge and native low-resource terminology, with top closed models dropping by over 20 percentage points on culturally specific tasks.

What Problem Does This Paper Address?

While multilingual LLM benchmarks are proliferating, the vast majority are simply automated translations of English benchmarks like MMLU. This introduces severe flaws: machine translation frequently bungles domain-specific terminology, and translated questions still test Western cultural, legal, and colloquial concepts rather than local knowledge.

Sinhala, an official language of Sri Lanka written in a non-Latin script, suffers from a severe scarcity of high-quality evaluation resources. To bridge this evaluation gap, the authors developed a benchmark sourced directly from primary native materials aligned with the Sri Lankan national curriculum, rather than relying on translated content.

Methods

The authors designed a standardized multiple-choice evaluation framework following MMLU conventions:

  • Native Question Extraction: Four qualified annotators extracted questions directly from official national and provincial government examination papers (GCE O-Level, GCE A-Level, and Grades 6–8) via the Ministry of Education's e-thaksalawa portal.
  • Cleaning and Deduplication: Questions were filtered to remove multimodal content (diagrams, images) and deduplicated using text embeddings (cosine similarity threshold > 0.95).
  • Evaluation Pipeline: Evaluated 26 open and closed LLMs across zero-shot and 3-shot prompting settings. Open models were evaluated via normalized token probabilities using 4-bit NF4 quantization, while closed models (GPT-4o, Claude) were queried via official APIs.
  • Controlled Diagnostics: The authors tested the effects of subject metadata in prompts, 4-option vs. 5-option choices, negation indicators, complex sub-option structures, and human naturalness ratings against translated alternatives.

Datasets

  • SinhalaMMLU (Introduced): 7,044 multiple-choice questions across 6 domains (Humanities, Social Science, STEM, Business Studies, Language, and Other) and 30 academic subjects, split into 3 difficulty tiers:
  • Easy (1,893 questions): Grades 6–8.
  • Medium (2,585 questions): Grades 9–11 (O-Level).
  • Hard (2,566 questions): Grades 12–13 (A-Level; featuring 5 answer choices).
  • Cultural Subset: A curated subset of 1,608 questions (22% of the dataset) requiring knowledge deeply embedded in Sri Lankan culture, history, language, and aesthetics.
  • GlobalMMLU-si (Baseline comparison): Used specifically to evaluate linguistic naturalness and translation quality against native STEM questions.

Key Insights and Ideas

  • Direct Translation Fails Technical Terminology: A human evaluation by Sri Lankan A-Level science students scored SinhalaMMLU STEM questions at 97.3% for linguistic naturalness, compared to just 71.07% for machine-translated GlobalMMLU-si. Translations often rely on transliterated phonetic loanwords (e.g., writing "plasmolysis" phonetically) rather than established technical Sinhala vocabulary (plasmacchēdanaya).
  • The "Culture Tax" is Substantial: Evaluating models exclusively on culturally specific questions triggered massive performance drops: Claude 3.5 Sonnet dropped 28.22% (from 67.65% to 39.43%) and GPT-4o dropped 23.83% (from 62.95% to 39.12%).
  • Instruction Tuning Hurts Few-Shot MCQ Performance: Base models showed modest gains when moving from zero-shot to 3-shot prompts, but instruction-tuned models (e.g., LLaMA-3.1-70B-chat) degraded in accuracy, illustrating that chat tuning can interfere with few-shot in-context learning.
  • Syntactic Complexity Stumbles Reasoning: Questions featuring negative phrasing (e.g., “not compatible”, “false”) or multi-statement combinations (e.g., “A and C only”) degraded accuracy across almost every model.

Key Results

  • Closed Models Lead: Claude 3.5 Sonnet achieved the highest zero-shot accuracy at 67.65%, followed by GPT-4o at 62.95%.
  • Open-Source Landscape: Qwen2.5-72B-chat was the highest-performing open model (41.18%), followed by Qwen2.5-32B-chat (36.47%). Smaller models (<8B parameters) largely hovered around 22–25%, barely exceeding random chance.
  • Domain Disparity: Models performed best on factual, standardized domains like Social Science (Claude: 77.55%) and Business Studies (73.12%), but lagged on culturally loaded or linguistically subtle domains like Language (62.37%) and STEM (61.40%).
  • Reducing Options: When collegiate-level hard questions were artificially reduced from 5 options to 4, performance jumped noticeably (+6.02% for Claude, +6.74% for Qwen-72B), though scores still remained lower than medium-difficulty questions.

Limitations and Future Work

  • Modality & Scope: The benchmark is strictly text-based MCQs, excluding diagrams, tables, and open-ended generative writing that characterize real-world examinations.
  • No Mathematics: Mathematics was omitted because Sri Lankan secondary curricula evaluate it through step-by-step problem-solving rather than MCQs.
  • Domain Skew: Because humanities and social science exam keys were more accessible, Humanities alone comprises 48% of the dataset.
  • Future Directions: The authors advocate for expanding into multimodal questions, primary- and professional-level exams, and fine-tuning language models with culturally aware Sinhala corpora.

Summary updated

Abstract

Large Language Models (LLMs) demonstrate impressive general knowledge and reasoning abilities, yet their evaluation has predominantly focused on global or anglocentric subjects, often neglecting low-resource languages and culturally specific content. While recent multilingual benchmarks attempt to bridge this gap, many rely on automatic translation, which can introduce errors and misrepresent the original cultural context. To address this, we introduce SinhalaMMLU, the first multiple-choice question answering benchmark designed specifically for Sinhala, a low-resource language. The dataset includes over 7,000 questions spanning secondary to collegiate education levels, aligned with the Sri Lankan national curriculum, and covers six domains and 30 subjects, encompassing both general academic topics and culturally grounded knowledge. We evaluate 26 LLMs on SinhalaMMLU and observe that, while Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies at 67% and 62% respectively, overall model performance remains limited. In particular, models struggle in culturally rich domains such as the Humanities, revealing substantial room for improvement in adapting LLMs to low-resource and culturally specific contexts.
BibTeX
@inproceedings{pramodya2025sinhalammlu,
  author = {Ashmari Pramodya and Nirasha Nelki and Heshan Shalinda and Chamila Liyanage and Yusuke Sakai and Randil Pushpananda and Ruvan Weerasinghe and Hidetaka Kamigaito and Taro Watanabe},
  title = {{SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=2tNPuosew5},
}

Loading related content...

Figure