IndT5: A Text-to-Text Transformer for 10 Indigenous Languages

El Moatez Billah Nagoudi, Wei-Rui Chen, Muhammad Abdul-Mageed, Hasan CavusogluNAACL 2021

TL;DR

This paper introduces IndT5, the first Transformer-based language model specifically pre-trained for ten Indigenous languages of the Americas, along with a new dataset called IndCorpus. By adapting the T5 text-to-text framework, IndT5 demonstrates improved machine translation performance for these low-resource languages, highlighting the potential of transfer learning and unified NLP approaches for linguistic diversity.

What Problem Does This Paper Address?

Natural Language Processing (NLP) has seen tremendous advancements with Transformer models, but these successes are largely concentrated on high-resource languages like English. Indigenous languages, on the other hand, often suffer from a severe lack of digital textual data, including parallel corpora needed for tasks like machine translation. This data scarcity, coupled with complex morphological systems and non-standardized orthographies, presents significant challenges for developing NLP tools for these languages. The paper aims to bridge this gap by creating resources and models for Indigenous languages, specifically focusing on machine translation from Spanish.

Methods

The core of this work is the development of IndT5, a Transformer model based on the T5 (Text-to-Text Transfer Transformer) architecture. T5 is notable for its unified approach, treating all NLP tasks as text-to-text problems (input text, output text). IndT5 leverages this by being pre-trained on a diverse corpus of Indigenous languages and Spanish.

The pre-training process involves a denoising objective, inspired by BERT, where the model learns to reconstruct corrupted text. This unsupervised approach allows IndT5 to learn general language representations without requiring labeled data. The authors then fine-tune IndT5 for the specific task of machine translation between Spanish and ten Indigenous languages.

Datasets

The paper introduces IndCorpus, a new dataset comprising approximately 1.17 GB of text (around 5.37 million sentences) for ten Indigenous languages and Spanish. This corpus was collected from Wikipedia and the Bible. Additionally, the authors utilize parallel data released for the AmericasNLP 2021 Shared Task on Open Machine Translation, which includes manually translated sentence pairs for eight language pairs (Spanish to target Indigenous languages).

Key Insights and Ideas

  • The T5 framework is highly adaptable: The text-to-text nature of T5 makes it a powerful tool for unifying various NLP tasks, including machine translation, even for low-resource languages.
  • Transfer learning is crucial for low-resource languages: Pre-training on a diverse corpus like IndCorpus significantly boosts performance on downstream tasks like machine translation, even when the target languages have very limited data.
  • Indigenous languages present unique challenges: Beyond data scarcity, complex morphology and orthographic variations require specialized attention, though this paper primarily focuses on data and model development.
  • Unified resources are vital: The creation of IndCorpus and IndT5 provides a valuable foundation for future research and development in Indigenous language NLP.

Key Results

The paper reports that IndT5 models, when fine-tuned for machine translation, outperform baselines across all tested language pairs. Specifically, fine-tuning with more data (Setting B, which incorporates 80% of the development set into training) generally leads to improvements in BLEU and ChrF++ scores. The authors note that further pre-training IndT5 on only the ten Indigenous languages (excluding Spanish) also yields better results, underscoring the importance of language-specific pre-training.

Limitations and Future Work

The authors acknowledge that despite improvements, machine translation for these Indigenous languages remains challenging due to the fundamental lack of parallel and even monolingual data. They suggest future work could involve integrating statistical MT methods to augment data and further exploring best hyperparameters for their neural models. They also plan to investigate the impact of limited data more broadly and explore other downstream tasks beyond machine translation.

Summary updated

BibTeX
@inproceedings{nagoudi2021indt,
  author = {El Moatez Billah Nagoudi and Wei-Rui Chen and Muhammad Abdul-Mageed and Hasan Cavusoglu},
  title = {{IndT5: A Text-to-Text Transformer for 10 Indigenous Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure