TL;DR
The authors introduce NaijaRC, the first human-annotated, multi-choice reading comprehension dataset for three major Nigerian languages: Hausa, Igbo, and Yorùbá. Sourced from real high-school African certifying exams, the dataset serves as a challenging benchmark revealing that while large language models like GPT-4 excel in English, smaller, specialized African-centric models often outperform them on native African languages.
What Problem Does This Paper Address?
While Natural Language Processing (NLP) has made massive leaps in reading comprehension (RC) for high-resource languages like English, under-resourced African languages have been largely left behind. Reading comprehension requires high-level reasoning, syntactic understanding, and real-world knowledge.
Prior attempts to build African RC datasets have relied heavily on automated translations from English datasets (such as the Belebele dataset). This approach fails to capture authentic cultural contexts, local idioms, and the unique structural nuances of native languages. To bridge this gap, the authors created a high-quality, human-verified evaluation dataset directly from native educational materials.
Methods
The researchers sourced reading comprehension passages, questions, and multiple-choice options from past editions of the West African Senior School Certificate Examination (WASSCE).
Because these historical exams only existed in printed formats, the team used Optical Character Recognition (OCR) tools like Google Lens and Apple OCR to digitize the text. Tonal languages like Igbo and Yorùbá rely heavily on diacritics (accents and markings on letters), which standard OCR tools frequently misinterpret. To ensure data integrity, native speakers and linguists manually cleaned, validated, and verified the entire dataset.
For evaluation, the authors established two primary baselines:
- Zero-Shot Cross-Lingual Transfer: Fine-tuning smaller, pre-trained encoder-only language models (PLMs)—specifically AfroXLMR-base, Serengeti, and OFA-768—on English training data from the Belebele dataset, and then evaluating them directly on NaijaRC.
- LLM Prompting: Directly prompting state-of-the-art large language models (GPT-3.5 Turbo and GPT-4 Turbo) by providing the context passage, question, and options to predict the correct answer.
Datasets
The paper utilizes two primary datasets:
- NaijaRC (New): Sourced from high-school exams for Hausa (
hau), Igbo (ibo), and Yorùbá (yor). The dataset splits consist of: - Hausa: 88 test questions
- Igbo: 88 test questions
- Yorùbá: 50 train, 25 dev, and 171 test questions
- Belebele: The English (
eng) training split (67,541 questions) was used to fine-tune the baseline PLMs.
Key Insights and Ideas
- The LLM Performance Drop: Massive LLMs exhibit a stark performance disparity between English and African languages. GPT-4 Turbo achieved an impressive 82.9% accuracy on English text but plummeted to an average of just 40.9% on the three Nigerian languages.
- Smaller, Specialized Models Win Locally: You don't always need billions of parameters. Smaller, specialized multilingual models tailored for African contexts frequently beat commercial LLMs. For instance, Serengeti outperformed all other models on Hausa, and OFA-768 took the crown for Igbo.
- The Diacritic Bottleneck: Curation of clean data for tonal African languages remains a major hurdle. Standard commercial OCR pipelines are not optimized for African language diacritics, necessitating expensive manual human intervention.
Key Results
As shown in the benchmark results, GPT-4 Turbo achieved the highest overall average accuracy (51.4%) across all four languages primarily due to its dominant English score. However, when looking strictly at the Nigerian languages, the zero-shot cross-lingual PLMs proved highly competitive:
- Hausa: Serengeti achieved the highest accuracy at 50.0% (outperforming GPT-4's 48.9%).
- Igbo: OFA-768 led with 48.9% accuracy (outperforming GPT-4's 40.9%).
- Yorùbá: AfroXLMR-base and GPT-4 Turbo tied for the top spot at 34.0%.
GPT-3.5 Turbo struggled significantly, yielding the lowest overall average accuracy at 40.0%.
Limitations and Future Work
The primary limitation of NaijaRC is its size; because it relies on manual verification and cleaning of physical exam papers, scaling the dataset is resource-intensive. Additionally, questions that relied on specific text formatting (like italics or underlined words to ask about context-specific semantics) had to be removed because current language models cannot process them effectively.
For future work, the authors plan to expand their evaluation into few-shot learning settings. Specifically, they aim to explore how providing models with a small set of examples (e.g., 50 passage-question-answer tuples) can improve the adaptation of existing reading comprehension models to African languages.
Summary updated