TL;DR
This paper introduces AfriTeVa, a family of encoder-decoder language models pretrained from scratch on a small corpus of 10 low-resource African languages. The work demonstrates that "small data" pretraining, previously successful for encoder-only models, can also be effective for sequence-to-sequence tasks like machine translation and text classification, offering a promising path for developing NLP capabilities for under-represented languages.
What Problem Does This Paper Address?
The incredible success of modern Natural Language Processing (NLP) is largely driven by large-scale pretraining of massive language models on vast amounts of text data. However, this paradigm creates a significant barrier for low-resource languages, particularly many African languages, which lack the extensive digital text corpora needed for such training. While encoder-only models like BERT have seen some success with "small data" pretraining approaches for these languages (e.g., AfriBERTa), extending this to more complex encoder-decoder models, crucial for generative tasks like translation, has remained an open question. This paper aims to bridge this gap by investigating whether the "small data" pretraining strategy can be effectively applied to sequence-to-sequence models for African languages.
Methods
The core of this work is the introduction of AfriTeVa, a family of sequence-to-sequence models based on the T5 architecture. The authors pretrained these models from scratch using a carefully curated corpus of 10 low-resource African languages. This "small data" approach contrasts with existing large multilingual models like mT5 and byT5, which, despite covering many languages, allocate only a small fraction of their training data to African languages.
The pretraining objective used is T5's "span-corruption" task, where contiguous spans of tokens are masked and replaced by a single sentinel token. The authors experimented with three model sizes: small (64M parameters), base (229M parameters), and large (745M parameters), mirroring T5's configurations. They also explored an additional variant that included English data during pretraining.
For evaluation, AfriTeVa models were fine-tuned on two downstream tasks:
- Text Classification: News title topic classification for Hausa and Yorùbá, framed as a text-to-text task where the decoder generates the class label.
- Machine Translation: Translation between English and seven African languages, using publicly available parallel corpora.
Datasets
The primary dataset used for pretraining AfriTeVa is the AfriBERTa corpus, which comprises approximately 1GB of text from 10 low-resource African languages: Afaan Oromoo, Amharic, Gahuza, Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya, and Yorùbá. An additional pretraining corpus was created by augmenting this with 1.5 million English sentences from the Common Crawl.
For fine-tuning, the following datasets were used:
- Text Classification: News title topic classification datasets for Hausa and Yorùbá (from Hedderich et al., 2020).
- Machine Translation: Publicly available parallel data for Hausa (10k sentences), Igbo (10k sentences), Yorùbá (10k sentences), Swahili (30k sentences), Luganda (7k sentences), Luo (7k sentences), and Pcm (8k sentences).
Key Insights and Ideas
- "Small Data" Pretraining Extends to Seq2Seq: The most significant finding is that the "small data" pretraining approach, which proved effective for encoder-only models like AfriBERTa, can indeed be successfully applied to encoder-decoder models like AfriTeVa for sequence-to-sequence tasks. This opens up new avenues for developing advanced NLP capabilities for languages with limited data.
- Competitive Performance with Limited Resources: AfriTeVa models, despite being pretrained on a fraction of the data used by large multilingual models, achieve competitive results on downstream tasks. This suggests that focused pretraining on relevant languages can be highly efficient.
- Tokenization Matters: The paper hints at the surprising effectiveness of byT5 on languages it wasn't explicitly trained on, suggesting that tokenization strategies might play a more significant role in cross-lingual transfer than currently understood. This is flagged as an area for future research.
- Scaling Benefits: Similar to larger models, increasing the size of AfriTeVa models (from small to large) leads to improved performance on both text classification and machine translation tasks, indicating that architectural scaling remains a relevant factor.
Key Results
- Machine Translation: AfriTeVa models, particularly the "base + En" variant, outperform mT5 on several African languages, despite mT5 being pretrained on a much larger and more diverse corpus. For instance, AfriTeVa base + En achieved an average BLEU score of 18.2 when translating from English to African languages, compared to mT5's 11.7. Scaling up AfriTeVa from base to large also yielded improvements, with the large model achieving an average BLEU of 13.7 for English-to-African language translation.
- Text Classification: AfriTeVa small outperforms mBERT and XLM-R on Hausa and Yorùbá text classification tasks, even with significantly fewer parameters. While AfriTeVa base was outperformed by AfriBERTa large, the models still performed better than mT5.
Limitations and Future Work
- Conclusive Evidence: The authors acknowledge that while their results are suggestive, they are not entirely conclusive. The "small data" approach shows promise, but it's difficult to definitively disentangle the effects of pretraining data size, model size, and initialization from larger, more established models.
- Dataset Limitations: A significant challenge highlighted is the scarcity of high-quality, diverse datasets for low-resource African languages, particularly for sequence-to-sequence tasks. Existing cross-lingual benchmarks often lack sufficient representation for many African languages.
- Future Work: The authors suggest several directions:
- Conducting a fairer comparison by pretraining models from scratch on larger, more diverse corpora (like AfriMT5's) to better understand the impact of initialization versus focused pretraining.
- Further investigating the role of tokenization in cross-lingual transfer, especially in light of byT5's performance.
- Continuing efforts to curate and create high-quality parallel datasets for African languages to drive further advances in machine translation and other NLP tasks.
- Exploring the effectiveness of AfriTeVa on a wider range of generative tasks beyond machine translation.
Summary updated