YORC: Yoruba Reading Comprehension dataset

Aremu Anuoluwapo, Jesujoba O. Alabi, David Ifeoluwa Adelani

Abstract

In this paper, we create YORC: a new multi-choice Yoruba Reading Comprehension dataset that is based on Yoruba high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.

Introduction

Natural Language Processing (NLP) has seen remarkable advancements in reading comprehension (RC), yet these gains are predominantly limited to high-resource languages like English. African languages, despite their large speaker bases, remain severely under-represented in benchmark datasets.

Current efforts to bridge this gap often rely on machine-translated datasets, such as Belebele. While useful, these translations frequently fail to capture linguistic nuances, cultural contexts, and the intricate tonal systems of languages like Yorùbá. There is a pressing need for "gold standard" data—content authored by native speakers for native speakers. This work addresses this gap by introducing NaijaRC, a human-annotated multi-choice reading comprehension benchmark for Hausa, Igbo, and Yorùbá, with a specific focus on the YORC (Yorùbá Reading Comprehension) subset.

Methodology

The creation of NaijaRC involved a rigorous "boots-on-the-ground" pipeline to ensure high data quality:

  1. Sourcing: The team collected past questions from the West African Senior School Certificate Examination (WASSCE), the standardized exit exam for high school students in West Africa.
  2. Digitization: As these exams were only available in print, Optical Character Recognition (OCR) tools were employed for digitization.
  3. Human Refinement: OCR often struggles with the specific diacritics (e.g., ẹ, ò, á) and special characters (e.g., ƙ, ɓ) essential to Nigerian languages. Native speakers and linguists manually corrected the digitized text to ensure semantic and tonal accuracy.
  4. Evaluation: Two primary model architectures were benchmarked: encoder-only models (fine-tuned on English data and tested in zero-shot settings) and Large Language Models (LLMs) evaluated via prompting.
Example of a multiple-choice question in Yorùbá from the dataset.

Dataset Description

The resulting NaijaRC dataset consists of multi-choice questions comprising a passage, a question, and four possible answers. The distribution across languages is as follows:

  • Hausa (hau): 88 test questions.
  • Igbo (ibo): 88 test questions.
  • Yorùbá (yor/YORC): 171 test questions.

The English Belebele dataset was utilized as a source for training and fine-tuning data, providing a baseline for cross-lingual transfer experiments.

Key Results

The benchmarking process revealed several critical insights into the current state of AI for African languages:

Accuracy scores for various models across Nigerian languages.
  • Specialization vs. Scale: While GPT-4 Turbo achieved the highest overall average (51.4%), this was heavily skewed by its superior performance in English (82.9%). In zero-shot settings for specific Nigerian languages, specialized models often outperformed the general-purpose LLM. For instance, Serengeti (an Africa-centric model) was the top performer for Hausa (50.0%), and OFA-768 led in Igbo (48.9%).
  • The Yorùbá Challenge: Yorùbá proved to be the most difficult language for all tested models, with a peak accuracy of only 34.0% (achieved by AfroXLMR-base). This underscores the complexity of tonal languages and the limitations of current cross-lingual transfer methods.
  • Cross-Lingual Transfer Hurdles: Accuracy scores generally fell between 30% and 50%, indicating that transferring reasoning capabilities from English to under-resourced languages remains a significant challenge.

Discussion and Limitations

The authors identify several constraints that define the scope of this work and point toward future research:

  • Dataset Scale: The reliance on manual human validation limits the dataset size to under 200 questions per language.
  • OCR Bottlenecks: The manual labor required to correct OCR errors for tonal languages is a major barrier to rapid scaling.
  • Visual Context: Questions requiring visual cues (like italics or underlining) were excluded, as current text-based models cannot yet process these formatting nuances effectively.

Future Work: The research team intends to explore few-shot settings, providing models with a small number of examples to determine if in-context learning can bridge the performance gap for Nigerian languages.

Summary updated

Abstract

In this paper, we create YORC: a new multi-choice Yoruba Reading Comprehension dataset that is based on Yoruba high-school reading comprehension examination. We provide baseline results by performing cross-lingual transfer using existing English RACE dataset based on a pre-trained encoder-only model. Additionally, we provide results by prompting large language models (LLMs) like GPT-4.
BibTeX
@inproceedings{anuoluwapo2024yorc,
  author = {Aremu Anuoluwapo and Jesujoba O. Alabi and David Ifeoluwa Adelani},
  title = {{YORC: Yoruba Reading Comprehension dataset}},
  booktitle = {AfricaNLP @ ICLR},
  year = {2024},
  url = {https://openreview.net/forum?id=xzi7gDeq5r},
}
Figure