BSC Submission to the AmericasNLP 2024 Shared Task

Javier Garcia Gilabert, Aleix Sant, Carlos Escolano, Francesca De Luca Fornaciari, Audrey Mash, Maite MeleroNAACL 2024

TL;DR

This paper presents the Barcelona Supercomputing Center's submission to the AmericasNLP 2024 Shared Task for Spanish-to-Quechua and Spanish-to-Guarani machine translation. They achieved state-of-the-art results by fine-tuning the NLLB-200 model, demonstrating that parameter-efficient fine-tuning with LoRA adapters can match full fine-tuning performance while significantly reducing computational cost.

What Problem Does This Paper Address?

Machine translation for indigenous languages of the Americas faces significant challenges due to a scarcity of textual resources. This lack of data makes it difficult to train high-quality translation models, especially for languages like Quechua and Guarani, which are primarily oral and have limited digital footprints. The AmericasNLP Shared Task aims to foster research and development in this area, pushing for better translation systems for these under-resourced languages. The paper addresses how to effectively fine-tune large pre-trained multilingual models for these specific low-resource language pairs, exploring the trade-offs between model size, data augmentation, and computational efficiency.

Methods

The core of the methodology involves fine-tuning the NLLB-200 model, a large multilingual neural machine translation model. They explore several approaches:

  • Baseline Fine-tuning: Fine-tuning separate NLLB-1.3B models for each language pair (Spanish-Quechua, Spanish-Guarani) using only bilingual data.
  • Multilingual Fine-tuning: Fine-tuning NLLB-1.3B with a mix of languages, including Portuguese and English, alongside Spanish data, to see if a richer linguistic context improves performance. They also investigate the impact of synthetic data generated by translating Spanish monolingual data.
  • Model Size Exploration: Comparing the performance of NLLB-1.3B and NLLB-3.3B models to understand if larger models inherently perform better for low-resource languages.
  • LoRA Fine-tuning: Employing Parameter-Efficient Fine-Tuning (PEFT) techniques, specifically Low-Rank Adaptation (LoRA). LoRA updates only a small subset of parameters by decomposing weight updates into low-rank matrices, significantly reducing trainable parameters and computational cost. They applied LoRA to the feed-forward and attention layers.
  • Inference Optimization: Conducting a grid search on beam size and repetition penalty to optimize translation quality during inference.

Datasets

The paper utilizes a variety of datasets for training and evaluation:

  • AmericasNLP 2024 Provided Data: Parallel data for Spanish-Quechua and Spanish-Guarani.
  • Jw300, MINEDU, dict_misc: For Quechua.
  • OPUS: Including GNOME, Mozilla-I10n, Tatoeba, Ubuntu, Wikimedia, and NLLB corpus.
  • FLoRes-200: A multiparallel evaluation dataset.
  • Hackathon-pln-es: Spanish to Quechua Ayacucho data.
  • Monolingual-Quechua-IIC: Used to generate synthetic Spanish translations.
  • TED2020: Spanish monolingual data used for synthetic data generation.
  • Helsinki University Data: Including OPUS resources.

Key Insights and Ideas

  • LoRA is a Game-Changer for Low-Resource MT: The most significant finding is that LoRA fine-tuning achieves performance comparable to full fine-tuning, even with drastically fewer trainable parameters (only 14.2% of the total). This is crucial for low-resource scenarios where computational power and data are limited.
  • Multilingual Data Benefits Quechua More: While multilingual data (including Portuguese and English) improved Spanish-to-Quechua translation, it did not significantly boost Spanish-to-Guarani performance. This suggests language family relationships and data availability play a key role.
  • Synthetic Data Boosts Quechua: The inclusion of synthetic target data (generated by translating Spanish monolingual data) positively impacted Spanish-to-Quechua translation quality.
  • Model Size Isn't Everything: Increasing model size from 1.3B to 3.3B parameters did not consistently yield superior results, especially when data is scarce. This highlights that efficient fine-tuning strategies are more critical than simply using larger models in such contexts.
  • Inference Tuning Matters: For Guarani, careful tuning of beam size and repetition penalty significantly improved translation quality, indicating that post-training optimization is vital.

Key Results

  • First Place Finish: The BSC submission achieved the highest ChrF++ scores and ranked first in both the Spanish-to-Quechua and Spanish-to-Guarani directions in the AmericasNLP 2024 Shared Task.
  • LoRA Performance: The best Spanish-to-Quechua result (38.21 ChrF++) was achieved using LoRA adapters with the NLLB-1.3B model. This system outperformed Sheffield's baseline by +4.2 points and Helsinki's baseline by +9.4 points.
  • Guarani Performance: For Spanish-to-Guarani, their best system achieved 38.93 ChrF++, outperforming Helsinki's baseline by +1.91 points and Sheffield's by +3.17 points.
  • Parameter Efficiency: LoRA fine-tuning with the NLLB-1.3B model required only 377,487,360 trainable parameters, a substantial reduction compared to the full NLLB-1.3B model (1,748,125,696 parameters).

Limitations and Future Work

  • Data Scarcity: The primary limitation remains the inherent scarcity of high-quality parallel data for these indigenous languages.
  • Model Size Impact: The paper notes that increasing model size did not consistently improve performance, possibly due to the limited amount of fine-tuning data. Future work could explore how larger models perform with more extensive data or more sophisticated fine-tuning techniques.
  • Guarani Data Mix: The Guarani data used in this work contained a mix of dialects, including "Jopara," which might affect translation consistency.
  • Inference Tuning Nuances: While beam search and repetition penalty were explored, further investigation into other inference parameters could potentially yield additional improvements.
  • Future Directions: The authors suggest that their findings on LoRA's effectiveness open avenues for more efficient development of MT systems for other low-resource languages. Further research could explore different PEFT methods or combinations of techniques.

Summary updated

BibTeX
@inproceedings{gilabert2024bsc,
  author = {Javier Garcia Gilabert and Aleix Sant and Carlos Escolano and Francesca De Luca Fornaciari and Audrey Mash and Maite Melero},
  title = {{BSC Submission to the AmericasNLP 2024 Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure