TL;DR
This paper introduces a new parallel dataset for Guinea-Bissau Creole (Kiriol) and investigates how to improve machine translation (MT) from religious texts to general domains. They demonstrate that even a small amount of target-domain data significantly boosts translation performance, highlighting the critical need for domain-specific data collection for low-resource languages.
What Problem Does This Paper Address?
Machine translation (MT) has seen remarkable progress, but many low-resource languages, particularly creoles, remain digitally underrepresented. A common issue is that the largest available parallel text corpora for these languages are often religious in nature (e.g., Bibles). While useful, training MT models solely on religious data leads to poor performance in general domains because religious texts have specialized vocabulary, theological concepts, and socio-cultural contexts that don't align with everyday language. This paper tackles the challenge of adapting MT models trained on religious data to perform better in general domains for Guinea-Bissau Creole (Kiriol), a language with limited digital resources.
Methods
The authors employ standard transformer-based encoder-decoder architectures for their MT models, implemented using the Eole toolkit. They train models from scratch, using byte-pair encoding (BPE) for tokenization. The core of their experimental approach involves:
- Dataset Creation: Compiling a new parallel corpus for Kiriol, English, and Portuguese.
- Domain Adaptation Experiments: Training models on religious data (Bible, Jehovah's Witnesses texts) and then evaluating their performance on a general domain test set. They then experiment with adding small amounts of general-domain data to these religious training sets to see if performance improves.
- Tokenization and Embedding Analysis: Investigating how different tokenization strategies (shared vs. separate) and embedding spaces (shared vs. separate) impact translation performance, particularly in relation to creole languages' morphological simplicity and lexical overlap with their lexifiers.
- Human Validation: Conducting a small-scale human evaluation to assess the quality of translations produced by different model configurations.
Datasets
- New Dataset: The paper introduces a new parallel dataset for Guinea-Bissau Creole (Kiriol) comprising approximately 40,000 parallel sentences. This dataset is primarily composed of:
- Religious Data: Bible verses (Almeida Revista e Corrigida for Portuguese, New International Version for English, Traduson Antigu for Kiriol) and semi-religious texts from Jehovah's Witnesses' Watchtower magazine.
- General Domain Data: Sentences from a Portuguese-Kiriol bilingual dictionary, designed to cover everyday topics.
- Lexical Data: Approximately 1,983 vocabulary items and 1,603 gloss sentences from the bilingual dictionary were also collected.
Key Insights and Ideas
- The Power of Small Target-Domain Data: Even a few hundred sentences of domain-specific data can dramatically improve MT performance on that domain, far more than simply increasing the amount of religious data. This suggests that targeted data collection, even on a small scale, is highly effective for low-resource MT.
- Creole Morphology and Tokenization: The morphological simplicity of creole languages like Kiriol can lead to less efficient tokenization when sharing tokenizers with morphologically complex languages (like Portuguese or English). This can disadvantage the creole in the MT model.
- Lexical Overlap Matters: A higher degree of lexical overlap between a creole and its lexifier language (like Kiriol and Portuguese) can be better leveraged through shared embeddings, leading to improved translation performance.
- Religious Data Limitations: Models trained solely on religious data perform poorly on general domain tasks, underscoring the need for domain adaptation strategies.
Key Results
- Adding just 300 general-domain dictionary sentences to a religious training set improved BLEU scores on the test set by 4.0 to 6.7 points.
- Oversampling 600 dictionary sentences five times yielded the best results on the test set, achieving an average BLEU of 11.9 across all language directions.
- Portuguese-Kiriol translation models generally performed better than other direction pairs, likely due to greater lexical overlap.
- Human validation showed a slight increase in accuracy and fluency when general-domain data was included, though the differences were less pronounced than BLEU scores suggested.
- Models trained with shared embeddings consistently outperformed those with separate embeddings, especially for Portuguese-Kiriol pairs.
Limitations and Future Work
- Generalizability of Findings: The findings regarding creole-lexifier relationships and their impact on MT might not generalize to all creole languages, as the formation and characteristics of creoles vary.
- BLEU Score Limitations: BLEU scores, while useful, are automated metrics and may not perfectly reflect human judgment of translation quality, especially when scores are low.
- From-Scratch vs. Fine-tuning: The study focused on training models from scratch. Future work could explore fine-tuning pre-trained multilingual models with Kiriol data, which might yield better results.
- Synthetic English Data: The English portion of the dictionary dataset was generated via automatic translation and manual checking, which could still impact its overall quality.
- Future Directions: The authors suggest exploring more advanced tokenization strategies, investigating how lexical overlap impacts other MT components (like attention mechanisms), and crucially, engaging with the Kiriol-speaking community to ensure that language technology development is accessible and beneficial.
Summary updated