TL;DR
The University of Sheffield team tackled the AmericasNLP 2023 Shared Task by fine-tuning and ensembling the NLLB-200 model for Spanish-to-Indigenous Language Machine Translation. Their approach achieved the highest average chrF score across all languages and secured first place for four of the eleven languages, demonstrating the effectiveness of leveraging large pre-trained models and diverse data sources for low-resource MT.
What Problem Does This Paper Address?
Machine translation (MT) for low-resource languages, particularly indigenous languages, remains a significant challenge. These languages often suffer from a lack of parallel data, complex linguistic structures (like polysynthesis and non-standardized word boundaries), and diverse orthographic variations. The AmericasNLP 2023 Shared Task specifically aimed to advance MT capabilities for eleven Spanish-Indigenous language pairs, many of which fall into this low-resource category. The paper addresses the problem of building effective MT systems for these languages by exploring how to best adapt and leverage existing state-of-the-art multilingual models.
Methods
The core of the Sheffield team's approach is the fine-tuning and ensembling of NLLB-200 (No Language Left Behind), a large, pre-trained multilingual neural machine translation model. They experimented with different sizes of NLLB-200, including 600M, 1.3B, and 3.3B parameters, primarily focusing on the 1.3B distilled version due to computational constraints.
Key methodological aspects include:
- Multilingual Fine-tuning: Models were trained on all 11 language pairs simultaneously, extending the embedding matrix to accommodate the new language tags.
- Data Augmentation: They incorporated various data sources beyond the provided shared task data, including constitutions, handbooks, news articles, and crucially, backtranslations generated from monolingual data.
- Ensembling: Multiple models were combined to improve overall performance, with specific ensemble strategies designed for different language groups or to leverage particular model strengths.
- Data Processing: Careful pre-processing was undertaken, including detokenization for specific languages and handling of punctuation and orthographic variations.
Datasets
The paper utilized several datasets for training and evaluation:
- AmericasNLP 2023 Data: Parallel datasets provided by the task organizers.
- Helsinki Data: Data from the University of Helsinki's submission to the 2021 AmericasNLP Shared Task, including translations of constitutions.
- REPUcs Data: Data collected for the REPUcs submission to the 2021 AmericasNLP Shared Task, featuring Quechua data from various sources (poems, stories, songs, vocabulary).
- NLLB Data: Datasets from Costa-jussà et al. (2022), including English-Aymara and English-Guarani examples.
- Bibles Data: Translations from the JHU Bible corpus (McCarthy et al., 2020).
- Backtranslations: Generated from monolingual data sourced from various previous works.
The paper notes that NLLB-200 itself was pre-trained on some of this data (e.g., Spanish-Aymara from GlobalVoices), but they argue its inclusion is still beneficial as NLLB-200 is not optimized for the specific task languages.
Key Insights and Ideas
- Leveraging Pre-trained Models is Key: The success of NLLB-200, even in its distilled forms, highlights the power of large pre-trained multilingual models for low-resource MT. Fine-tuning these models significantly outperforms baselines and even random initialization.
- Multilingual Training Benefits Transfer: Training models on all language pairs simultaneously, rather than in isolation, demonstrably improves performance, suggesting beneficial transfer learning across related and even unrelated indigenous languages.
- Backtranslations Help, But Selectively: While backtranslations are a valuable data augmentation technique, their impact was most pronounced on languages with more available monolingual data (gn and shp).
- Data Scarcity Can Lead to Surprising Results: The language with the least training data (Chatino, czn) surprisingly yielded the highest chrF score for the team's best submission on the test set, suggesting that model performance isn't always directly proportional to data volume in complex low-resource scenarios.
Key Results
- Overall Performance: The Sheffield submission achieved the highest average chrF score across all languages on the test set (30.5), outperforming the second-best average by 1.1 points.
- Top Rankings: They ranked first in four of the eleven languages (aym, czn, quy, shp) and achieved top-3 rankings for all languages.
- Dev Set Improvements: On the dev set, their best submission outperformed the baseline by an average of 11% chrF across all languages, with substantial gains for Aymara, Guarani, and Quechua.
- Model Size vs. Performance: While NLLB-3.3B generally performed better, the 1.3B distilled version offered a good balance of performance and computational feasibility, enabling more extensive experimentation.
- Zero-shot Capabilities: The models demonstrated a decent ability to perform zero-shot translation, retaining up to 75% of their performance on unseen language pairs (e.g., quy-shp, aym-shp, gn-shp).
Limitations and Future Work
- Data Overlap: The authors acknowledge that NLLB-200 was pre-trained on some of the data used for fine-tuning. While they believe this is still beneficial, it's a factor to consider.
- Bibles Data Ineffectiveness: Surprisingly, incorporating Bible data did not lead to performance improvements for most languages, except for gn and oto. This suggests that the domain and style of the Bible text may not be optimal for general-purpose MT in these contexts.
- Oto Performance: For the oto language, their performance was comparable to the baseline, indicating potential challenges or limitations in adapting the model for this specific language.
- Future Work: The paper implicitly suggests further exploration into optimal ensembling strategies, more targeted data augmentation techniques, and potentially investigating the linguistic properties that facilitate cross-lingual transfer. The zero-shot results also open avenues for research into improving this capability.
Summary updated