The REPU CS’ Spanish–Quechua Submission to the AmericasNLP 2021 Shared Task on Open Machine Translation

Oscar MorenoNAACL 2021

TL;DR

This paper details REPUcs' submission to the AmericasNLP 2021 Shared Task for Spanish-Quechua machine translation. They achieved first place in one track and third in another by creating new parallel corpora and employing transfer learning from large Spanish-English datasets. This work highlights the challenges and potential of machine translation for low-resource indigenous languages.

What Problem Does This Paper Address?

Machine translation (MT) systems typically perform best for high-resource languages with abundant parallel data. However, many languages, particularly indigenous ones like Quechua, suffer from a severe lack of digital text, making it difficult to train effective MT models. This scarcity is exacerbated by the fact that many of these languages are primarily spoken, with limited written documentation. The AmericasNLP 2021 Shared Task specifically targeted this challenge by focusing on the Spanish-Quechua language pair, aiming to advance MT capabilities for such low-resource scenarios.

Methods

The REPUcs team employed a two-pronged approach:

  • Data Augmentation: They manually collected and aligned new parallel data from various sources, including poems, lyrics, lexicons, and handbooks. This was crucial for supplementing the limited existing datasets.
  • Transfer Learning: They leveraged pre-trained NMT models. Specifically, they first trained a Spanish-English model using the large EuroParl dataset. This pre-trained model was then fine-tuned on Spanish-Quechua data. To further enhance performance, they followed the strategy of sharing the source language and subword vocabulary, inspired by previous work on low-resource MT.
  • Subword Segmentation: Recognizing Quechua's polysynthetic nature (where words can carry a lot of grammatical information), they used Byte-Pair Encoding (BPE) with a large vocabulary size (32,000). They trained the BPE model on a combination of Spanish, English, and Quechua data, upsampling the Quechua data to ensure a more uniform distribution.

Datasets

  • AmericasNLP Shared Task Datasets:
    • JW300: Religious domain texts (Spanish-Quechua, Spanish-English). Noted for misaligned sentences and misspellings.
  • MINEDU: Short, open-domain sentences from the Peruvian Ministry of Education dictionary (Spanish-Quechua).
  • Dict_misc: Dictionary entries and samples (Spanish-Quechua).
  • Development Set: 994 sentences from Spanish and Quechua (quy).
  • Newly Introduced Corpora:
    • WebMisc: Aligned poems and stories from a website encouraging Quechua Ayacucho learning.
  • Handbook: Aligned translations from a Quechua Ayacucho handbook.
  • Lexicon: Aligned dictionary entries and samples.
  • Constitution & Regulation: Parallel corpora extracted from official documents for Quechua Cusco (quz), though not used in the final submission due to time constraints.
  • Pre-training Dataset:
    • EuroParl: 1.9 million Spanish-English sentences.

Key Insights and Ideas

  • Data is King (Even for Transfer Learning): While transfer learning from high-resource languages is beneficial, the creation of new, domain-relevant parallel data for Spanish-Quechua significantly boosted performance.
  • Subword Segmentation is Crucial for Polysynthetic Languages: Quechua's agglutinative nature means standard word-level translation struggles. Effective subword segmentation is essential for capturing morphological nuances.
  • Domain Mismatch is a Real Challenge: The paper highlights a significant domain mismatch (less than 50% token overlap) between the training and development sets for Quechua, underscoring the difficulty of generalizing MT models in low-resource settings.
  • chrF is a Better Metric for Agglutinative Languages: The authors note that chrF, which evaluates character n-grams, is more suitable than BLEU for agglutinative languages like Quechua, as it can better capture morphological variations.

Key Results

The REPUcs submission achieved:

  • Track 1 (Development set used for training): 3rd place with a BLEU score of 2.91 and chrF of 0.358.
  • Track 2 (Development set not used for training): 1st place with a BLEU score of 3.63 and chrF of 0.343.

The transfer learning approach showed a considerable improvement over the baseline, with a +0.56 increase in BLEU and +0.007 in chrF when using the provided datasets. Combining the new corpora with transfer learning yielded the best results, demonstrating the effectiveness of their strategy.

Limitations and Future Work

  • Quechua Cusco Data: The parallel corpora extracted from official documents for Quechua Cusco (quz) were not used in the final submission due to time constraints but are suggested as a valuable resource for future work.
  • Agglutinative Nature: The authors acknowledge that translating Quechua remains challenging due to its agglutinative nature and the difficulty of effective subword segmentation. They suggest that further research into subword segmentation methods is warranted.
  • Low BLEU Scores: While the results are competitive, the overall BLEU scores are noted as being low. The authors attribute this to the inherent difficulty of decoding agglutinative languages and suggest that character-level metrics like chrF are more informative.

Summary updated

BibTeX
@inproceedings{moreno2021repu,
  author = {Oscar Moreno},
  title = {{The REPU CS’ Spanish–Quechua Submission to the AmericasNLP 2021 Shared Task on Open Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure