Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in English and other high-resource languages. However, their performance in low-resource African languages remains a significant "blind spot" in AI research. While existing evaluations for low-resource languages (LRLs) often focus on shallow tasks like machine translation or basic text classification, high-stakes applications—such as health advice or educational tutoring—require models to handle knowledge-intensive tasks.
This paper introduces Uhura, a first-of-its-kind benchmark designed to evaluate scientific reasoning and truthfulness across six typologically diverse African languages. By providing a high-quality, human-verified dataset, Uhura reveals a stark performance divide between high-resource and low-resource settings, even in state-of-the-art models.
The Uhura Benchmark
Uhura is built upon two foundational English datasets: ARC-Easy, which consists of multiple-choice science questions for grades 3-9, and TruthfulQA, a safety benchmark testing truthfulness across 38 categories such as health, law, and politics. These datasets were translated into six African languages representing different linguistic families:
- Amharic (Semitic)
- Hausa (Chadic)
- Northern Sotho/Sepedi (Bantu)
- Swahili (Bantu)
- Yoruba (Niger-Congo)
- Zulu (Bantu)
To ensure linguistic and cultural accuracy, the researchers collaborated with professional translators from the Masakhane NLP community. Unlike automated translation pipelines, this human-in-the-loop approach utilized a custom annotation platform with text-to-speech features to verify the naturalness of the translated text. The resulting dataset is multi-way parallel, allowing for direct performance comparisons across languages.
Experimental Results and Analysis
The study evaluated a wide range of models, including proprietary giants (GPT-4o, o1-preview, Claude 3.5 Sonnet) and open-source contenders (Llama-3.1, Gemma-2, AfroLlama). The researchers also tested five different prompt templates and the impact of In-Context Learning (ICL) to ensure the robustness of the results.
The "English Premium" and Capability Chasm
A primary finding is the persistent "English Premium": every model performed significantly better in English than in any African language. For GPT-4o, the performance gap exceeded 17 percentage points, suggesting that even "multilingual" models are heavily optimized for English logic and data.
Furthermore, a significant "capability chasm" exists between proprietary and open-source models. On the ARC-Easy benchmark, the best open-source model (Gemma-2-27b-it) achieved 42.6% accuracy, while the best closed-source model (o1-preview) reached 92.4%.
Linguistic Inequality and Truthfulness
Performance was not uniform across the six languages. Swahili consistently saw higher scores, likely due to its better representation in training data, while Amharic and Northern Sotho often lagged behind.
Truthfulness remains a major challenge. Models were much more prone to generating false claims or superstitions when prompted in African languages. On TruthfulQA, even the top-performing models struggled to cross the 70% accuracy mark in LRLs, highlighting the risks of deploying these models for factual inquiries in these regions.
Discussion and Limitations
The authors identify several hurdles in current benchmarking practices. A significant issue is cultural mismatch; because the source datasets are Western-centric (e.g., questions about U.S. geography or flag etiquette), they may not fully capture local knowledge or common misconceptions specific to African contexts. Additionally, while the dataset is substantial (approximately 1,200 questions for ARC-Easy and 817 for TruthfulQA per language), some technical terms lack direct equivalents in LRLs, leading to potential translation nuances.
Conclusion
Uhura highlights the urgent need for more inclusive AI development. While proprietary models like OpenAI’s o1-preview lead the pack—achieving an average of 92.4% on Uhura-ARC-Easy—the massive performance gap compared to English (99.5%) underscores the work remaining. Future efforts will focus on expanding the benchmark to more languages and developing "native" benchmarks that originate from African contexts rather than relying on translations of Western data.
Summary updated