NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages

Anuoluwapo Aremu, Jesujoba O. Alabi, David Ifeoluwa Adelani, Aguobi, Nkechinyere F., Muhammad, Shamsuddeen Hassan, Adelani, David Ifeoluwa

Abstract

In this paper, we create NaijaRC: a new multi-choice Reading Comprehension dataset for three native Nigeria languages that is based on high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE and Belebele training dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.

Introduction

African languages are significantly under-resourced in the field of Natural Language Processing (NLP). While progress has been made in tasks like translation and named entity recognition, Reading Comprehension (RC)—the ability of a machine to understand a passage and answer complex questions—remains a major challenge. Existing RC datasets for African languages, such as Belebele, are often machine-translated from English, which can lead to the loss of cultural context and critical tonal markers. To address this, we introduce NaijaRC, the first human-annotated multi-choice RC dataset for Hausa, Igbo, and Yorùbá, sourced directly from high-school exit examinations.

The NaijaRC Dataset

NaijaRC is derived from the West African Senior School Certificate Examination (WASSCE). By using real-world academic exams, we ensure the dataset tests high-level reasoning rather than simple pattern matching. The dataset covers three major Nigerian languages: Hausa (hau), Igbo (ibo), and Yorùbá (yor).

Dataset statistics and model accuracy across different languages.

The creation of NaijaRC involved significant data preservation efforts. Since the source material was print-only, we utilized Optical Character Recognition (OCR) tools. However, the complex diacritics used in Nigerian languages to denote tone (e.g., ẹ, è, á) often resulted in OCR errors that required manual correction by native speakers.

Example of the complex diacritics and multiple-choice format found in the Yorùbá portion of the dataset.

Methodology

We evaluated AI capabilities using two primary methods:

  1. Zero-Shot Cross-Lingual Transfer: Encoder-only models (e.g., AfroXLMR, Serengeti) pre-trained on multilingual text were fine-tuned on English datasets (Belebele and RACE) and then tested on NaijaRC without target-language training.
  2. LLM Prompting: State-of-the-art models like GPT-3.5 and GPT-4 were prompted to select the correct answer given a passage and question.

Key Results and Insights

Our evaluation revealed several critical findings regarding the performance of general-purpose versus specialized models:

  • Specialization vs. Scale: While GPT-4 Turbo achieved the highest average accuracy (51.4%), specialized models often outperformed it in specific languages. For instance, Serengeti—a model tailored for African languages—surpassed GPT-4 in Hausa (50.0% vs 48.9%).
  • The Scale Paradox: Bigger models are not always better for low-resource, tonal languages. GPT-3.5 Turbo struggled significantly, averaging only 40.0% accuracy, which was lower than several smaller, specialized encoder models like AfroXLMR-base.
  • The Diacritic Challenge: The authors highlight that most standard NLP models still struggle with the orthography of Nigerian languages. Manual labor was required to fix OCR errors and remove questions relying on visual formatting cues that models cannot yet process.

Conclusion

NaijaRC provides a rigorous benchmark for evaluating NLP models on native Nigerian languages. The results highlight a persistent "digital divide," where even advanced general-purpose LLMs lag behind specialized models or show significantly lower performance compared to English. Future work will explore few-shot learning to further bridge this gap and improve model understanding of Nigerian linguistic structures.

Summary updated

Abstract

In this paper, we create NaijaRC: a new multi-choice Reading Comprehension dataset for three native Nigeria languages that is based on high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE and Belebele training dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.
BibTeX
@inproceedings{aremu2024naijarc,
  author = {Anuoluwapo Aremu and Jesujoba O. Alabi and David Ifeoluwa Adelani and Aguobi, Nkechinyere F. and Muhammad, Shamsuddeen Hassan and Adelani, David Ifeoluwa},
  title = {{NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages}},
  booktitle = {AfricaNLP @ ICLR},
  year = {2024},
  url = {https://openreview.net/forum?id=xzi7gDeq5r},
}
Figure