AGE: Amharic, Ge’ez and English Parallel Dataset

Henok Biadglign Ademtew, Mikiyas Girma BirboACL 2024

TL;DR

This paper introduces AGE, a novel tripartite parallel dataset for Amharic, Ge'ez, and English, addressing the critical lack of resources for low-resource languages in NLP. They demonstrate its utility by fine-tuning the NLLB-200 model, achieving promising results, especially for Ge'ez-to-English translation, and making the dataset publicly available to foster further research.

What Problem Does This Paper Address?

Natural Language Processing (NLP) has seen tremendous advancements, largely driven by the availability of massive parallel datasets for high-resource languages like English. However, many languages, particularly those in Africa, remain severely underrepresented. This "low-resource" status hinders the development and application of modern NLP technologies, including machine translation (MT), for these languages. Ge'ez, an ancient Ethiopian language with significant cultural and religious importance, and Amharic, Ethiopia's official working language, are prime examples. Existing MT systems and large pre-trained models often lack support for Ge'ez, creating a barrier for its use in digital contexts and for its preservation. This paper aims to bridge this gap by creating a valuable resource for these languages.

Methods

The core of this work is the creation of the AGE (Amharic, Ge'ez, and English) parallel dataset. The authors employed a multi-stage process:

  1. Data Collection: They gathered existing parallel sentences from various sources, including religious texts (like "Kufale" - The Book of Jubilees), the Open Siddur Project, YouVersion, Ethiopic Bible, and Awde Mehrets. A crucial part of their contribution is the creation of 1,000 new Ge'ez-centered sentences sourced from news and novels, which were then translated by human experts.
  2. Data Pre-processing: The collected data underwent rigorous cleaning to address textual inconsistencies. This involved removing URLs, hashtags, and duplicate sentences, normalizing Amharic characters, converting English characters to lowercase, and handling null values.
  3. Model Fine-tuning: To evaluate the dataset's utility and establish baseline performance, the authors fine-tuned the NLLB-200 600M model. This is a dense transformer model distilled from the larger NLLB-200, chosen for its reduced computational requirements. They split their dataset into training (80%), development (10%), and testing (10%) sets and used standard fine-tuning procedures with a learning rate of 5e-5 and a batch size of 4 per device.

Datasets

The primary dataset introduced is AGE: Amharic, Ge'ez, and English Parallel Dataset. This tripartite dataset includes:

  • 17,585 Amharic-Ge'ez sentence pairs.
  • 18,676 Ge'ez-English sentence pairs.
  • The paper also mentions using existing parallel corpora from previous research for comparison (e.g., Mulugeta, 2015; Abate et al., 2019; Getachew and Yayeh, 2023).

Key Insights and Ideas

  • The critical need for curated, high-quality data: The paper highlights that even with large pre-trained models, performance on low-resource languages is severely hampered by data scarcity and quality. Their manual creation of Ge'ez-centered sentences is a testament to this.
  • Ge'ez-to-English translation shows surprising strength: Despite Ge'ez not being part of NLLB-200's pre-training, the model achieved significantly higher BLEU scores for Ge'ez-to-English translation compared to Amharic-Ge'ez directions. This suggests that fine-tuning on even limited, high-quality data can unlock substantial potential.
  • Morphological complexity is a major challenge: The lower BLEU scores for Amharic-Ge'ez and Ge'ez-Amharic are attributed to the inherent morphological complexity of Semitic languages like Ge'ez, making it difficult for models to capture nuances and inflectional ambiguities.
  • Publicly available resources are crucial for progress: The authors emphasize that releasing their dataset and models is vital for the broader NLP community to build upon, fostering collaborative efforts for low-resource languages.

Key Results

The fine-tuned NLLB-200 600M model achieved the following baseline BLEU scores:

  • Amharic-Ge'ez: 9.39
  • Ge'ez-Amharic: 12.29
  • English-Ge'ez: 12.87
  • Ge'ez-English: 30.66

The significantly higher scores for Ge'ez-to-English translation (30.66) are noted as particularly encouraging. The paper also includes a loss graph (Figure 2) showing convergence of training loss for all language pairs over 10,000 steps, indicating the model is approaching optimal performance.

Limitations and Future Work

  • Data Quantity: While the AGE dataset is a significant contribution, the authors acknowledge that increasing the quantity and diversity of the data is essential for further improvements.
  • Bidirectional English-Amharic Translation: The primary focus was on Ge'ez, so bidirectional English-Amharic translation was not trained.
  • Future Work: The authors plan to increase the dataset's size and diversity and incorporate additional Ge'ez data sources that were not included in this initial release. They also intend to release their models and dataset to the public.

Summary updated

BibTeX
@inproceedings{ademtew2024age,
  author = {Henok Biadglign Ademtew and Mikiyas Girma Birbo},
  title = {{AGE: Amharic, Ge’ez and English Parallel Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure