Abstract
Large Language Models (LLMs) demonstrate impressive general knowledge and reasoning abilities, yet their evaluation has predominantly focused on global or anglocentric subjects, often neglecting low-resource languages and culturally specific content. While recent multilingual benchmarks attempt to bridge this gap, many rely on automatic translation, which can introduce errors and misrepresent the original cultural context. To address this, we introduce SinhalaMMLU, the first multiple-choice question answering benchmark designed specifically for Sinhala, a low-resource language. The dataset includes over 7,000 questions spanning secondary to collegiate education levels, aligned with the Sri Lankan national curriculum, and covers six domains and 30 subjects, encompassing both general academic topics and culturally grounded knowledge. We evaluate 26 LLMs on SinhalaMMLU and observe that, while Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies at 67% and 62% respectively, overall model performance remains limited. In particular, models struggle in culturally rich domains such as the Humanities, revealing substantial room for improvement in adapting LLMs to low-resource and culturally specific contexts.
Introduction
While Large Language Models (LLMs) like GPT-4o and Claude 3.5 Sonnet exhibit remarkable capabilities in English, their performance in low-resource languages remains largely unquantified. For the 17 million Sinhala speakers in Sri Lanka, existing AI benchmarks are often inadequate, relying on machine-translated versions of English tests. These "GlobalMMLU" variants frequently suffer from technical inaccuracies and a lack of cultural relevance, failing to capture the nuances of local knowledge.
To bridge this "evaluation gap," researchers have introduced SinhalaMMLU, the first native, culturally grounded benchmark for the Sinhala language. By utilizing questions written by and for Sri Lankans, this benchmark provides a more authentic measure of an LLM's understanding of the language and its specific cultural context.
The SinhalaMMLU Dataset
The SinhalaMMLU dataset consists of 7,044 multiple-choice questions meticulously sourced from Sri Lankan national and provincial exam papers, ranging from Grade 6 to Grade 13 (A-Levels).
The questions are categorized into six primary domains: Humanities, Social Science, STEM, Language, Business Studies, and "Other." As illustrated in the domain distribution, the Humanities domain comprises nearly half of the dataset (48%), reflecting the importance of culturally specific knowledge—such as Buddhism, local history, and traditional arts—in the Sri Lankan education system.
Evaluation and Results
The researchers evaluated 26 diverse LLMs, including both proprietary giants and open-source models. The results highlight a significant performance gap between state-of-the-art models and the baseline of human-level understanding in Sinhala.
Key Findings:
- Top Performers: Claude 3.5 Sonnet achieved the highest average accuracy at 67.65%, followed by GPT-4o at 62.95%. Among open-source models, Qwen2.5-72B-chat led the pack, though it trailed significantly behind the top proprietary models.
- The Difficulty Gap: As shown in the performance chart, accuracy declines sharply as the academic level increases. On "Hard" (A-Level) questions, many models struggle to perform significantly better than random chance.
- Cultural Blind Spots: Models generally performed better in "global" subjects like Geography or Civics. However, accuracy dropped by up to 28 percentage points in the "Cultural" subset, which includes indigenous dancing, Sinhala literature, and local history.
- Naturalness vs. Translation: Human annotators rated SinhalaMMLU questions at 97.3% naturalness, compared to only 71.07% for translated benchmarks. This confirms that native datasets provide a much fairer and more accurate assessment of linguistic competence.
Discussion and Limitations
The study reveals that while scaling model size generally improves performance, current LLMs still possess significant weaknesses in logical reasoning (particularly with negations) and cultural depth within the Sinhala script.
Current limitations of the benchmark include the exclusion of image-based questions and complex mathematical problems due to formatting constraints. Future iterations of SinhalaMMLU aim to incorporate primary-level content and generative tasks to further explore the conversational and creative boundaries of these models in a local context.
Conclusion
SinhalaMMLU serves as a critical tool for evaluating and improving AI for the Sinhala-speaking community. By highlighting the "culture gap" and the limitations of translated benchmarks, this work underscores the necessity of developing native, high-quality datasets for low-resource languages to ensure global AI equity.
Summary updated