Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages

Edward Bayes, Israel Abebe Azime, Jesujoba O. Alabi, Jonas Kgomo, Tyna Eloundou, Elizabeth Proehl, Kai Chen, Imaan Khadir, Naome A. Etori, Shamsuddeen Hassan Muhammad, Choice Mpanza, Igneciah Pocia Thete, Dietrich Klakow, David Ifeoluwa AdelaniarXiv 2024 · DOI · arXiv

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in English and other high-resource languages. However, their performance in low-resource African languages remains a significant "blind spot" in AI research. While existing evaluations for low-resource languages (LRLs) often focus on shallow tasks like machine translation or basic text classification, high-stakes applications—such as health advice or educational tutoring—require models to handle knowledge-intensive tasks.

This paper introduces Uhura, a first-of-its-kind benchmark designed to evaluate scientific reasoning and truthfulness across six typologically diverse African languages. By providing a high-quality, human-verified dataset, Uhura reveals a stark performance divide between high-resource and low-resource settings, even in state-of-the-art models.

The Uhura Benchmark

Uhura is built upon two foundational English datasets: ARC-Easy, which consists of multiple-choice science questions for grades 3-9, and TruthfulQA, a safety benchmark testing truthfulness across 38 categories such as health, law, and politics. These datasets were translated into six African languages representing different linguistic families:

  • Amharic (Semitic)
  • Hausa (Chadic)
  • Northern Sotho/Sepedi (Bantu)
  • Swahili (Bantu)
  • Yoruba (Niger-Congo)
  • Zulu (Bantu)

To ensure linguistic and cultural accuracy, the researchers collaborated with professional translators from the Masakhane NLP community. Unlike automated translation pipelines, this human-in-the-loop approach utilized a custom annotation platform with text-to-speech features to verify the naturalness of the translated text. The resulting dataset is multi-way parallel, allowing for direct performance comparisons across languages.

Experimental Results and Analysis

The study evaluated a wide range of models, including proprietary giants (GPT-4o, o1-preview, Claude 3.5 Sonnet) and open-source contenders (Llama-3.1, Gemma-2, AfroLlama). The researchers also tested five different prompt templates and the impact of In-Context Learning (ICL) to ensure the robustness of the results.

The "English Premium" and Capability Chasm

A primary finding is the persistent "English Premium": every model performed significantly better in English than in any African language. For GPT-4o, the performance gap exceeded 17 percentage points, suggesting that even "multilingual" models are heavily optimized for English logic and data.

Furthermore, a significant "capability chasm" exists between proprietary and open-source models. On the ARC-Easy benchmark, the best open-source model (Gemma-2-27b-it) achieved 42.6% accuracy, while the best closed-source model (o1-preview) reached 92.4%.

Linguistic Inequality and Truthfulness

Performance was not uniform across the six languages. Swahili consistently saw higher scores, likely due to its better representation in training data, while Amharic and Northern Sotho often lagged behind.

Truthfulness remains a major challenge. Models were much more prone to generating false claims or superstitions when prompted in African languages. On TruthfulQA, even the top-performing models struggled to cross the 70% accuracy mark in LRLs, highlighting the risks of deploying these models for factual inquiries in these regions.

Discussion and Limitations

The authors identify several hurdles in current benchmarking practices. A significant issue is cultural mismatch; because the source datasets are Western-centric (e.g., questions about U.S. geography or flag etiquette), they may not fully capture local knowledge or common misconceptions specific to African contexts. Additionally, while the dataset is substantial (approximately 1,200 questions for ARC-Easy and 817 for TruthfulQA per language), some technical terms lack direct equivalents in LRLs, leading to potential translation nuances.

Conclusion

Uhura highlights the urgent need for more inclusive AI development. While proprietary models like OpenAI’s o1-preview lead the pack—achieving an average of 92.4% on Uhura-ARC-Easy—the massive performance gap compared to English (99.5%) underscores the work remaining. Future efforts will focus on expanding the benchmark to more languages and developing "native" benchmarks that originate from African contexts rather than relying on translations of Western data.

Summary updated

Abstract

Evaluations of Large Language Models (LLMs) on knowledge-intensive tasks and factual accuracy often focus on high-resource languages primarily because datasets for low-resource languages (LRLs) are scarce. In this paper, we present Uhura -- a new benchmark that focuses on two tasks in six typologically-diverse African languages, created via human translation of existing English benchmarks. The first dataset, Uhura-ARC-Easy, is composed of multiple-choice science questions. The second, Uhura-TruthfulQA, is a safety benchmark testing the truthfulness of models on topics including health, law, finance, and politics. We highlight the challenges creating benchmarks with highly technical content for LRLs and outline mitigation strategies. Our evaluation reveals a significant performance gap between proprietary models such as GPT-4o and o1-preview, and Claude models, and open-source models like Meta's LLaMA and Google's Gemma. Additionally, all models perform better in English than in African languages. These results indicate that LMs struggle with answering scientific questions and are more prone to generating false claims in low-resource African languages. Our findings underscore the necessity for continuous improvement of multilingual LM capabilities in LRL settings to ensure safe and reliable use in real-world contexts. We open-source the Uhura Benchmark and Uhura Platform to foster further research and development in NLP for LRLs.
BibTeX
@misc{bayes2024uhura,
  author = {Edward Bayes and Israel Abebe Azime and Jesujoba O. Alabi and Jonas Kgomo and Tyna Eloundou and Elizabeth Proehl and Kai Chen and Imaan Khadir and Naome A. Etori and Shamsuddeen Hassan Muhammad and Choice Mpanza and Igneciah Pocia Thete and Dietrich Klakow and David Ifeoluwa Adelani},
  title = {{Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages}},
  year = {2024},
  eprint = {2412.00948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.00948},
}
Figure