TL;DR
This paper details NRC-CNRC's participation in the AmericasNLP 2021 shared task, focusing on machine translation for low-resource Indigenous languages of the Americas. They achieved strong results by employing multilingual pretraining, ensembling, and careful finetuning strategies, demonstrating the effectiveness of these techniques even with limited data.
What Problem Does This Paper Address?
The paper tackles the significant challenge of building effective machine translation (MT) systems for low-resource languages, specifically focusing on Indigenous languages of the Americas. These languages often suffer from a scarcity of parallel data, making it difficult to train traditional MT models. Furthermore, these languages can exhibit considerable dialectal and orthographic variation, and the available data might not match the domain of interest for potential applications. The AmericasNLP shared task aimed to push the boundaries of MT for these languages, and this paper explores how standard NMT techniques can be adapted and applied in such challenging settings.
Methods
The NRC-CNRC team employed a multi-pronged approach to tackle the low-resource MT problem:
- Multilingual Pretraining: They leveraged the power of multilingual NMT models, training a single model on multiple language pairs. This allows the model to learn shared linguistic patterns and generalize better, especially when data for individual languages is scarce.
- Ensembling: To further improve robustness and performance, they ensembled multiple NMT systems. This involves combining the predictions of several models to arrive at a final output, often leading to better overall results than any single model.
- Finetuning: They explored various finetuning strategies:
- Language Finetuning: After initial multilingual training, they specialized the models for each target language by continuing training on that language's specific data.
- Development Finetuning: Recognizing that the development and test sets might differ in domain or dialect from the training data, they also finetuned models using parts of the development data itself. This helps the model adapt to the target domain.
- BPE-Dropout: They incorporated BPE-dropout as a regularization technique. This method generates multiple versions of the training corpus with BPE applied differently, helping the model become more robust to variations and potentially improving performance on noisy or varied data.
- Translation Memory Baseline: As a strong baseline, they also implemented a translation memory (TM) system. This approach leverages existing parallel data (in this case, the development set) to find similar source sentences and provide their translations as a starting point for human post-editing.
Datasets
The primary datasets used were those provided for the AmericasNLP 2021 shared task. These included parallel data for translating from Spanish into:
- Wixárika (hch)
- Nahuatl (nah)
- Rarámuri (tar)
- Guaraní (gn)
The paper notes that these datasets vary in size, dialect, and orthographic consistency, and often present a domain mismatch with the development and test sets.
Key Insights and Ideas
- Multilingualism is Key: The study strongly suggests that multilingual NMT systems, even those combining languages from different families, consistently outperform monolingual approaches in low-resource settings.
- Finetuning Pays Off: Both language-specific finetuning and finetuning on development data significantly improved translation quality, especially when dealing with domain mismatches.
- BPE-Dropout's Utility: BPE-dropout proved beneficial for most languages, helping to mitigate data sparsity and potentially improving robustness to orthographic variation.
- Translation Memory as a Strong Baseline: For low-resource scenarios with domain mismatches, a translation memory approach, even using only a fraction of the development data, can serve as a surprisingly strong baseline, outperforming MT systems trained without development data.
- Domain Mismatch is a Major Hurdle: The paper highlights that the significant domain mismatch between training and development/test sets is a primary reason for the extremely low automatic metric scores observed in this task.
Key Results
The NRC-CNRC systems achieved competitive results, placing them as the second-ranking team in the "with-development-set" group and among the top teams in the "no-development-set" cluster.
- Their best system (S.0), an ensemble incorporating multilingual training, finetuning (including on development data), and BPE-dropout, achieved the highest scores.
- For Wixárika and Rarámuri, the translation memory submission proved to be a particularly strong baseline.
- While automatic metric scores (CHRF) were generally low across all languages, the finetuning strategies demonstrably improved performance. For instance, systems finetuned on development data (e.g., S.2) showed substantially higher scores than those trained without it (e.g., S.1).
Limitations and Future Work
- Data Scarcity: The fundamental limitation remains the extremely small size of the parallel data for these languages. The authors suggest that additional data collection, especially in the target domain, is one of the most effective ways to improve quality.
- Domain Mismatch: The significant difference in domain between training and development/test sets posed a major challenge, leading to low automatic scores.
- Human Evaluation: The authors acknowledge that automatic metrics like CHRF and BLEU are not perfectly correlated and that human evaluation is crucial to truly assess the usefulness and quality of the generated translations, especially for downstream tasks.
- Catastrophic Forgetting: During finetuning, there's a risk of "catastrophic forgetting," where the model loses previously learned information. The authors suggest exploring modified learning rates for finetuning as a future direction.
- Architecture Exploration: While they experimented with some architectural variations, they note that a wider search, as explored in other work, could yield further improvements. They also suggest exploring a smaller number of decoder heads as a promising avenue.
- Language-Specific Pre/Post-processing: While they applied some language-specific pre- and post-processing, they acknowledge that more sophisticated approaches might be beneficial.
Summary updated