Peru is Multilingual, Its Machine Translation Should Be Too?

Arturo OncevayNAACL 2021

TL;DR

This paper explores the potential of multilingual Neural Machine Translation (NMT) for low-resource indigenous languages in Peru. By training a single NMT model for multiple languages simultaneously, the authors demonstrate improved translation quality for languages like Aymara, Ashaninka, and Shipibo-Konibo when translating to/from Spanish, outperforming traditional pairwise models.

What Problem Does This Paper Address?

Peru is a linguistically rich country with 47 indigenous languages, many of which are endangered and lack sufficient digital resources for traditional machine translation (MT) development. While large, high-resource languages benefit greatly from NMT, low-resource languages face significant challenges due to data scarcity. This paper investigates whether a multilingual NMT approach, leveraging shared linguistic features and pre-trained models, can effectively bridge this resource gap and improve translation quality for these underrepresented languages.

Methods

The authors employ a multilingual NMT strategy, building upon several key techniques:

  • Multilingual Subword Segmentation: They utilize a unigram language model-based SentencePiece tokenizer with a vocabulary size of 32,000 to create a shared subword segmentation model across all target languages. This approach aims to handle the agglutinative and polysynthetic nature of these languages by preserving roots and affixes.
  • Pre-training: Two Transformer-base NMT models are pre-trained on a large English-Spanish parallel corpus in both directions. This leverages existing high-resource data to build a strong foundation for transfer learning.
  • Multilingual Fine-tuning: The pre-trained models are then fine-tuned on the parallel data for the target Peruvian languages (Aymara, Ashaninka, Quechua, and Shipibo-Konibo) paired with Spanish. Uniform sampling is used to balance the training data from different language pairs.
  • Back-translation (BT): To further leverage monolingual data, back-translation is applied. This involves translating monolingual data from the target languages into Spanish using the trained multilingual model, and then using this synthetic parallel data for further training.
  • Tagged Back-translation (BT[t]): To mitigate potential issues with noisy synthetic data, a special tag is introduced to signal to the model when it's processing back-translated sentences.

Datasets

The study utilizes datasets from the AmericasNLP Shared Task, including:

  • Southern Quechua (quy): Parallel data from dictionaries and Jehovah Witnesses, aligned with English.
  • Aymara (aym): Parallel and monolingual data extracted from news websites (Global Voices), aligned with English.
  • Shipibo-Konibo (shp): Parallel data from dictionaries, educational materials, and language learning flashcards, along with monolingual data from educational books.
  • Ashaninka (cni): Parallel data from dictionaries, laws, and books, plus monolingual corpus.

For pre-training, the authors use:

  • English-Spanish datasets: EuroParl (1.7M sentences) and NewsCommentary-v8 (174k sentences).

Additionally, for out-of-domain evaluation, they use:

  • Quechua (Kallpa): A small evaluation set (100 sentences) from a magazine.
  • Shipibo-Konibo (Kirika): A new evaluation set (200 sentences) of short traditional stories.

Key Insights and Ideas

  • Multilingualism as a Solution: The core idea is that leveraging the shared linguistic space of a multilingual country through a single NMT model can be more effective than isolated, pairwise models for low-resource languages.
  • Agglutinative Languages are Challenging: The highly agglutinative and polysynthetic nature of these indigenous languages poses a significant challenge for standard subword segmentation, often leading to "rare words." The unigram SentencePiece approach is a step towards addressing this.
  • Back-translation's Double-Edged Sword: While back-translation is a common technique to boost performance with monolingual data, it can sometimes degrade performance if the synthetic data is too noisy or dominates the training set. The paper explores tagged back-translation as a way to mitigate this.
  • Quechua as a "Higher-Resource" Case: Quechua, despite being an indigenous language, has more available data than the others. Interestingly, the pairwise model performed best for Quechua, suggesting that for languages with slightly more resources, the benefits of multilingual models might be less pronounced or require different strategies.

Key Results

  • Improved Performance for Low-Resource Languages: Multilingual NMT models (especially with tagged back-translation) significantly outperformed pairwise baselines for Aymara, Ashaninka, and Shipibo-Konibo in both Spanish-to-many and many-to-Spanish directions, as measured by BLEU and chrF scores.
  • Back-translation Nuances: While back-translation generally improved scores, the "tagged" version (BT[t]) often yielded better or more stable results than standard BT, suggesting its utility in handling synthetic data.
  • Quechua Outlier: For Quechua, the pairwise model achieved the best BLEU scores, indicating that the multilingual approach did not universally benefit all languages in the experiment.
  • Out-of-Domain Robustness: Fine-tuning the multilingual models on out-of-domain data showed promising results, with models retaining performance on the original test sets while adapting to new domains, suggesting good generalization capabilities.

Limitations and Future Work

  • Quechua Performance: The paper acknowledges that the pairwise model outperformed the multilingual one for Quechua. Future work could explore strategies to better integrate higher-resource languages into multilingual models or investigate domain-specific adaptations.
  • Back-translation Quality: The authors note that the performance degradation with back-translation might stem from noisy synthetic data. Future research could focus on more robust denoising or online data selection strategies.
  • Evaluation Metrics: The paper highlights that BLEU, a word-level metric, may not fully capture the nuances of highly agglutinative languages. The authors suggest that character-level metrics (like chrF, which they used) are important, and further human evaluation would be beneficial.
  • Zero-Shot Translation: A key future goal is to train the first many-to-many multilingual model for these indigenous languages, enabling zero-shot translation between any pair of languages in the model.
  • Domain Adaptation: The authors suggest using domain tags to signal more information to the model, supporting better fine-tuning across diverse datasets.

Summary updated

BibTeX
@inproceedings{oncevay2021peru,
  author = {Arturo Oncevay},
  title = {{Peru is Multilingual, Its Machine Translation Should Be Too?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure