Fostering Digital Inclusion for Low-Resource Nigerian Languages: A Case Study of Igbo and Nigerian Pidgin

Ebelechukwu Nwafor, Minh Phuc NguyenNAACL 2025

TL;DR

This paper tackles the challenge of poor machine translation performance for low-resource Nigerian languages like Igbo and Nigerian Pidgin. By creating and releasing new benchmark datasets, the authors demonstrate that fine-tuning existing state-of-the-art models significantly outperforms large language models like GPT-4 for these languages, paving the way for more inclusive AI.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) and, specifically, machine translation (MT), has seen tremendous advancements, largely driven by sophisticated deep learning architectures like transformers and the availability of vast amounts of digital text. However, these advancements have disproportionately benefited "high-resource" languages (e.g., English, German, Spanish) that have abundant digital data. "Low-resource" languages (LRLs), which are less studied and digitized, are left behind, leading to a significant digital divide. This is particularly true for many African languages. This paper focuses on two such languages, Igbo and Nigerian Pidgin, highlighting the critical need for high-quality datasets and improved translation models to foster digital inclusion and reduce linguistic bias in AI.

Methods

The core of this research involves creating and utilizing benchmark datasets to evaluate and improve machine translation for Igbo and Nigerian Pidgin. The authors employed a two-pronged approach for data annotation:

  • Crowdsourced Human Annotation: They leveraged Amazon Mechanical Turk, hiring native Nigerian speakers to translate sentences. Strict quality control measures were implemented, including random checks against AI translations (GPT-3.5 and GPT-4) to disqualify any AI-assisted work and scrutinizing translations for accuracy and speed.
  • LLM Annotation: To supplement human efforts and due to cost constraints, they also used GPT-3.5 and GPT-4 to generate translations for a portion of the dataset.

The researchers then used these newly created datasets, along with existing open-source benchmark datasets, to fine-tune three prominent multilingual translation models: NLLB, ByT5, and DeltaLM. The goal was to compare the performance of these fine-tuned models against each other and against large language models (LLMs) like GPT-4.

Datasets

The paper introduces two key contributions in terms of datasets:

  • Igbo-English and Nigerian Pidgin-English Benchmark Datasets: These datasets were created by combining data scraped from online sources (BBC Igbo, BBC Pidgin, Ted Talk, Voice of Nigeria, Naijalingo) and then meticulously annotated by native speakers and LLMs. The initial collection yielded over 2 million monolingual sentences.
  • Leveraging External Datasets: For comparative analysis and fine-tuning, the researchers also utilized several existing open-source datasets:
  • Flores200: A large multilingual dataset covering 204 languages.
  • MafandMT: A dataset focused on 11 African languages.
  • JW300: A broad multilingual dataset with over 109 million sentences.
  • IgboNLP: A smaller Igbo-English dataset.

Key Insights and Ideas

  • Fine-tuning is Crucial for LRLs: The study strongly suggests that fine-tuning pre-trained models on language-specific data is essential for achieving good performance in low-resource language translation, often surpassing the capabilities of general-purpose LLMs.
  • LLMs Still Lag Behind Specialized Models for LRLs: Despite their impressive general capabilities, LLMs like GPT-4 did not perform as well as fine-tuned, specialized models on the Igbo-English translation tasks, especially when compared to models like NLLB.
  • Dataset Quality Matters: The creation of high-quality, human-annotated parallel datasets is a significant bottleneck for LRLs, and this research demonstrates how such datasets can dramatically improve translation performance.
  • Promoting Linguistic Diversity in AI: The paper underscores the broader impact of this work in fostering digital inclusion by addressing the underrepresentation of many languages in AI technologies.

Key Results

  • Fine-tuned models showed significant improvements in BLEU scores compared to their original versions. On average, NLLB's score increased by 44%, ByT5 by 17.84%, and DeltaLM by 25.64%.
  • The original DeltaLM and ByT5 models produced poor outputs with BLEU scores below 1% before fine-tuning, indicating a struggle with the limited Igbo training data and multilingual interference.
  • Fine-tuned NLLB achieved the highest performance on the Flores200 and AmazonMTurk datasets.
  • Fine-tuned DeltaLM and ByT5 performed well on the MafandMT and JW300 datasets.
  • Overall, the fine-tuned NLLB model achieved the highest average BLEU score across the tested datasets, outperforming GPT-4 in most cases.

Limitations and Future Work

  • LLM Annotation Fraction: Due to cost, only a fraction of the original monolingual dataset was annotated using LLMs.
  • Model Selection: The study focused on three specific models (NLLB, ByT5, DeltaLM) and did not explore all available state-of-the-art models.
  • Future Work: The authors suggest that their created datasets will serve as a foundation for future research, enabling further development of AI models for low-resource languages. They also highlight the potential of LLMs in low-resource settings and the need for more equitable AI applications across languages.

Summary updated

BibTeX
@inproceedings{nwafor2025fostering,
  author = {Ebelechukwu Nwafor and Minh Phuc Nguyen},
  title = {{Fostering Digital Inclusion for Low-Resource Nigerian Languages: A Case Study of Igbo and Nigerian Pidgin}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure