TL;DR
This paper presents two approaches for low-resource Indigenous language machine translation: fine-tuning the powerful NLLB-200 model and training a novel Mamba-based architecture from scratch. The fine-tuned NLLB-200 model achieved state-of-the-art results on several languages, demonstrating the effectiveness of leveraging large pre-trained models for under-resourced languages.
What Problem Does This Paper Address?
Machine translation (MT) systems have made significant strides, but they often struggle with low-resource languages, particularly those with limited digital data. This paper tackles the challenge of building effective MT systems for 11 Indigenous languages of the Americas, which are often characterized by scarce parallel corpora. The authors aim to improve translation quality for these languages, bridging the gap between widely spoken languages and those with less digital presence.
Methods
The researchers employed two primary methods:
- Fine-tuning NLLB-200: They fine-tuned the 600M distilled variant of the NLLB-200 model. Recognizing that the token distribution in low-resource languages might differ significantly from the languages NLLB-200 was originally trained on, they chose to fine-tune all parameters, including embeddings, to better adapt to these new distributions. They introduced additional language tokens for languages not already represented in the model. Training was conducted in stages, with earlier stages using noisier synthetic data and later stages focusing on higher-quality data.
- Mamba State Space Model: They also trained a neural network from scratch using the Mamba architecture. This model consists of repeating Mamba layers and a language model head. For training, they prepended target language tokens to source sentences and added start/end of sentence tokens. This approach was trained for a longer duration, with an initial phase on all data followed by extended training on higher-quality data.
Datasets
The study utilized several data sources:
- Organizer-provided Data: This included training and development sets for the 11 target languages, along with supplemental translation data from Spanish and English.
- HelsinkiNLP Data: Data from the 2023 HelsinkiNLP submission, sourced from OPUS, FLORES-200, and the JHU Bible corpus, covering various domains.
- Tatoeba Translation Challenge: Parallel data for Guarani, Nahuatl, and Quechua, as well as pivot translations generated from non-Spanish parallel data.
- Pivot Translations: The authors generated additional parallel data using machine translation systems, translating data from languages like English, Esperanto, French, and others into Spanish.
The data was organized into three stages, with rigorous duplicate filtering applied across these stages to ensure data quality.
Key Insights and Ideas
- Leveraging Pre-trained Models is Key: Fine-tuning a large, pre-trained model like NLLB-200 proved highly effective for low-resource languages, outperforming baseline systems and achieving state-of-the-art results on several language pairs. This highlights the power of transfer learning in MT.
- Full Fine-tuning is Beneficial: The decision to fine-tune all parameters of NLLB-200, including embeddings, was crucial. This suggests that adapting the entire model, rather than just the top layers, is necessary when dealing with significantly different language characteristics.
- Mamba Shows Promise, but Needs More Training: While the Mamba model trained from scratch showed poor performance at the time of submission, the authors believe this is due to undertraining. They suggest that continued training could lead to competitive results, indicating the potential of this novel architecture for MT.
- Data Staging and Filtering are Crucial: The multi-stage training approach and meticulous duplicate filtering were essential for managing and maximizing the utility of the available data, especially the noisier synthetic data.
Key Results
The fine-tuned NLLB-200 models achieved the best chrF++ scores on 4 out of the 11 target languages (Aymara, Shipibo-Konibo, and Rarámuri). They also improved upon baseline scores for two languages. Specifically, the NLLB+FT(v2) and NLLB+FT(v4) models achieved average chrF++ scores of 22.17 and 23.32 respectively, with NLLB+FT(v4) being the second-best overall submission. The Mamba model, while underperforming at submission time, is expected to improve with further training.
Limitations and Future Work
- Dialectal Variation: The authors acknowledge that the performance of their systems might be limited when encountering dialectal or orthographic variations not present in the training data. For instance, the Quechua data used was specific to the Ayacucho variant.
- Mamba Undertraining: The Mamba model's poor performance is attributed to insufficient training. The authors plan to continue experimenting with this architecture to explore its full potential.
- Future Work: Continued training and experimentation with the Mamba architecture is a primary future direction. Further investigation into handling dialectal variations could also improve system robustness.
Summary updated