Enhancing Spanish-Quechua Machine Translation with Pre-Trained Models and Diverse Data Sources: LCT-EHU at AmericasNLP Shared Task

Nouman Ahmed, Natalia Flechas Manrique, Antonije PetrovićACL 2023

TL;DR

This paper tackles the challenge of low-resource machine translation for Spanish-Quechua by exploring various pre-trained models and data augmentation techniques. They found that fine-tuning a Spanish-Finnish model, leveraging the morphological similarities between Finnish and Quechua, yielded the best results, outperforming models trained on Spanish-English data.

What Problem Does This Paper Address?

Machine translation (MT) systems typically perform best when trained on large amounts of parallel data (text in one language aligned with its translation in another). However, many languages, particularly indigenous ones like Quechua, suffer from a severe lack of such data, making it difficult to build effective MT systems. This paper focuses on the Spanish-Quechua language pair, a prime example of a low-resource scenario, aiming to improve translation quality by creatively leveraging existing resources and techniques.

Methods

The LCT-EHU team explored several strategies:

  • Data Augmentation: They collected and aligned new parallel corpora from diverse sources, including legal documents from Ecuador, "The Little Prince," and the UN Declaration of Human Rights. They also experimented with synthetic data generation techniques like back-translation and a "copied corpus" approach (reusing monolingual data as if it were parallel).
  • Pre-trained Models: A core part of their approach involved fine-tuning existing pre-trained MT models. They compared two main setups:
  • Spanish-English: Using a high-resource Spanish-English model as a starting point, leveraging its strong Spanish encoder.
  • Spanish-Finnish: This was their key innovation. They hypothesized that Finnish, an agglutinative language like Quechua, would provide a better linguistic foundation for fine-tuning.
  • Model Training: Models were trained for a fixed number of epochs (20), with evaluation performed periodically. The best model for each setup was selected based on the chrF score on a development set.

Datasets

The authors utilized a combination of datasets:

  • Original AmericasNLP 2023 Shared Task Data: This included JW300 (religious texts), MINEDU (dictionary), and Dict_misc (dictionary entries).
  • Additional Resources from Previous Years: Data from the Helsinki-NLP team (Peruvian Constitution, Bolivian Constitution, Tatoeba, Bible) and REPUcs-AmericasNLP2021 team (Web Misc, Lexicon, Handbook, Peruvian Constitution, Amazon Parliament regulations).
  • Newly Collected Parallel Data:
    • Legal documents: UN Human Rights Declaration, Constitution of Ecuador, Ley Soberania Alimentaria, Ley Consumo Drogas, Ley Organica Alimentacion.
  • Literature: "The Little Prince."
    • Additional Cuzco Quechua data from a previous toolkit.
  • Historical texts: "Comentarios Reales," "Nueva Cronica y Buen Gobierno," and "Cronica del Peru."
  • Monolingual Data: cc100 and Llamacha datasets were used for the copied corpus approach.

Key Insights and Ideas

  • Finnish as a Bridge: The most significant finding is the effectiveness of using a Spanish-Finnish pre-trained model. The morphological similarity between Finnish and Quechua proved to be a crucial factor, allowing the model to transfer linguistic knowledge more effectively than a Spanish-English model.
  • Data Quality Matters: While augmenting data is generally beneficial, the authors observed that low-quality synthetic data (specifically the "copied corpus") could actually degrade performance. This highlights the importance of careful data curation and quality assessment.
  • Domain Specificity: The alignment of legal documents for Spanish-Quechua was found to be more accurate than for literary texts like "The Little Prince." This suggests that the inherent structure and vocabulary of legal texts might be easier to align.

Key Results

  • The highest-performing model on the development set achieved a chrF score of 37.70. This model was a Spanish-Finnish model fine-tuned on the original training data, augmented with data from previous years' submissions (including Ayacucho Quechua).
  • On the test set, the highest performing model achieved a chrF score of 38.59. This model used the same Spanish-Finnish base but was trained on the original training data, data from previous years' submissions (excluding Cuzco Quechua), and the newly aligned data.
  • The Spanish-English fine-tuned models performed reasonably well but were consistently outperformed by the Spanish-Finnish models.
  • The "copied corpus" approach, while intended to boost data, led to a decrease in chrF scores across experiments.

Limitations and Future Work

  • Data Quality of Synthetic Data: The paper acknowledges that the "copied corpus" approach led to performance degradation, suggesting that further investigation into the quality and impact of synthetic data is needed.
  • Quechua Varieties: The paper uses "que" as a macrolanguage identifier for Quechua varieties, acknowledging the linguistic diversity within the language. Future work could explore specific varieties more deeply.
  • Model Availability: The authors have made their source code and collected data publicly available, which is a positive step for future research. They also mention that all their models are available on Huggingface, encouraging further experimentation.

Summary updated

BibTeX
@inproceedings{ahmed2023enhancing,
  author = {Nouman Ahmed and Natalia Flechas Manrique and Antonije Petrović},
  title = {{Enhancing Spanish-Quechua Machine Translation with Pre-Trained Models and Diverse Data Sources: LCT-EHU at AmericasNLP Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure