Abstract
In this paper, we create NaijaRC: a new multi-choice Reading Comprehension dataset for three native Nigeria languages that is based on high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE and Belebele training dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.
Introduction
African languages are significantly under-resourced in the field of Natural Language Processing (NLP). While progress has been made in tasks like translation and named entity recognition, Reading Comprehension (RC)—the ability of a machine to understand a passage and answer complex questions—remains a major challenge. Existing RC datasets for African languages, such as Belebele, are often machine-translated from English, which can lead to the loss of cultural context and critical tonal markers. To address this, we introduce NaijaRC, the first human-annotated multi-choice RC dataset for Hausa, Igbo, and Yorùbá, sourced directly from high-school exit examinations.
The NaijaRC Dataset
NaijaRC is derived from the West African Senior School Certificate Examination (WASSCE). By using real-world academic exams, we ensure the dataset tests high-level reasoning rather than simple pattern matching. The dataset covers three major Nigerian languages: Hausa (hau), Igbo (ibo), and Yorùbá (yor).
The creation of NaijaRC involved significant data preservation efforts. Since the source material was print-only, we utilized Optical Character Recognition (OCR) tools. However, the complex diacritics used in Nigerian languages to denote tone (e.g., ẹ, è, á) often resulted in OCR errors that required manual correction by native speakers.
Methodology
We evaluated AI capabilities using two primary methods:
- Zero-Shot Cross-Lingual Transfer: Encoder-only models (e.g., AfroXLMR, Serengeti) pre-trained on multilingual text were fine-tuned on English datasets (Belebele and RACE) and then tested on NaijaRC without target-language training.
- LLM Prompting: State-of-the-art models like GPT-3.5 and GPT-4 were prompted to select the correct answer given a passage and question.
Key Results and Insights
Our evaluation revealed several critical findings regarding the performance of general-purpose versus specialized models:
- Specialization vs. Scale: While GPT-4 Turbo achieved the highest average accuracy (51.4%), specialized models often outperformed it in specific languages. For instance, Serengeti—a model tailored for African languages—surpassed GPT-4 in Hausa (50.0% vs 48.9%).
- The Scale Paradox: Bigger models are not always better for low-resource, tonal languages. GPT-3.5 Turbo struggled significantly, averaging only 40.0% accuracy, which was lower than several smaller, specialized encoder models like AfroXLMR-base.
- The Diacritic Challenge: The authors highlight that most standard NLP models still struggle with the orthography of Nigerian languages. Manual labor was required to fix OCR errors and remove questions relying on visual formatting cues that models cannot yet process.
Conclusion
NaijaRC provides a rigorous benchmark for evaluating NLP models on native Nigerian languages. The results highlight a persistent "digital divide," where even advanced general-purpose LLMs lag behind specialized models or show significantly lower performance compared to English. Future work will explore few-shot learning to further bridge this gap and improve model understanding of Nigerian linguistic structures.
Summary updated