Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas

Ona De Gibert, Robert Pugh, Ali Marashian, Raul Vazquez, Abteen Ebrahimi, Pavel Denisov, Enora Rice, Edward Gow-Smith, Juan Prieto, Melissa Robles, Rubén Manrique, Oscar Moreno, Angel Lino, Rolando Coto-Solano, Aldo Alvarez, Marvin Agüero-Torales, John E. Ortega, Luis Chiruzzo, Arturo Oncevay, Shruti Rijhwani, Katharina von der Wense, Manuel MagerNAACL 2025

TL;DR

This paper presents the outcomes of the AmericasNLP 2025 Shared Tasks, focusing on advancing Natural Language Processing (NLP) for Indigenous languages of the Americas. The research tackles critical challenges in machine translation, educational material creation, and evaluation metrics, highlighting the persistent difficulties and promising progress in this under-resourced domain.

What Problem Does This Paper Address?

The rapid advancements in NLP, largely driven by Large Language Models (LLMs), have predominantly benefited high-resource languages like English and French. Indigenous languages, often characterized by limited digital text, complex morphology, and non-standardized orthographies, are left behind. This disparity hinders their preservation, revitalization, and the inclusion of their speakers in the digital age. The AmericasNLP workshop aims to bridge this gap by fostering research and developing technologies specifically for these languages.

Methods

The paper details three shared tasks (STs) that address distinct challenges:

  • ST1: Machine Translation (MT) for Low-Resource Languages: This task focused on translating between Spanish and 13 Indigenous languages. It introduced new translation directions (e.g., into Spanish) and languages. The primary approach involved fine-tuning large multilingual pre-trained models like NLLB-200-3.3B. Some teams also explored prompt engineering with LLMs like GPT-4 and LLaMA, often incorporating external linguistic knowledge.
  • ST2: Morphological Adaptation for Educational Material Creation: This task aimed to automatically transform sentences to generate grammar exercises for language learners. It involved modifying grammatical features like tense, aspect, and mood. Approaches included prompt engineering with LLMs and, for Nahuatl, a rule-based system.
  • ST3: Developing Metrics for MT in Indigenous Languages: This task addressed the inadequacy of standard MT evaluation metrics (like BLEU) for morphologically rich and non-standardized languages. The focus was on developing metrics that better capture semantic and fluency aspects. Approaches included fine-tuning language-agnostic models (LaBSE) and developing custom metrics, alongside traditional overlap-based metrics like ChrF++.

Datasets

The shared tasks utilized existing datasets and introduced new ones where necessary:

  • ST1: Leveraged data from previous AmericasNLP shared tasks, augmented with new parallel corpora for Awajun and Wayuunaiki. Datasets included Spanish-Indigenous language pairs, with some derived from sources like the Bible, legal texts, and dictionaries.
  • ST2: Used existing datasets for Bribri, Guarani, and Maya, and introduced a new dataset for an endangered variety of Nahuatl (Western Sierra Puebla Nahuatl). The Nahuatl data was sourced from a corpus of grammatical example sentences, verified by native speakers.
  • ST3: Employed sentence pairs from AmericasNLP 2024 MT ST submissions, annotated for semantics and fluency by bilingual speakers.

Key Insights and Ideas

  • Multilingual Pre-trained Models Remain Strong Baselines: For machine translation (ST1), fine-tuning large multilingual models like NLLB-200-3.3B proved to be a very strong baseline, often outperforming more complex LLM-based approaches. This suggests that for low-resource languages, leveraging existing broad linguistic knowledge is highly effective.
  • LLMs Show Promise for Specific Directions and Tasks: While not always beating fine-tuned baselines, LLMs demonstrated utility, particularly for translation into Spanish (ST1) and for generating educational materials (ST2). Prompt engineering with LLMs, especially when combined with linguistic knowledge, showed potential.
  • Morphological Complexity is a Major Hurdle: The paper repeatedly highlights that the agglutinative and polysynthetic nature of many Indigenous languages poses significant challenges for both MT and metric development. This complexity often requires specialized handling beyond standard NLP techniques.
  • Knowledge-Based Methods Still Have Value: For tasks involving complex grammatical transformations (ST2), knowledge-based or rule-based approaches, particularly for languages like Nahuatl, showed competitive or superior performance compared to LLM-based methods. This suggests a hybrid approach might be optimal.
  • Standard Metrics Fall Short: The limitations of metrics like BLEU for Indigenous languages are re-emphasized, driving the need for new evaluation methods that better capture semantic and fluency aspects in morphologically rich languages.

Key Results

  • ST1: The GMU team achieved the best overall performance in ST1, particularly in Track 2 (Indigenous language to Spanish). The baseline model was difficult to surpass, indicating the effectiveness of broad multilingual training.
  • ST2: The paper notes that while LLMs were effective for some languages, results on Nahuatl were challenging, with a rule-based system performing best. This highlights the difficulty of adapting LLMs to highly complex or under-represented language varieties.
  • ST3: The Raava_5 system achieved the best Pearson correlation for ST3, indicating strong performance in capturing both semantics and fluency. This suggests that combining various linguistic and computational features can lead to effective MT evaluation metrics for Indigenous languages.

Limitations and Future Work

  • Data Scarcity: The fundamental challenge of limited parallel and monolingual data for many Indigenous languages persists, impacting the performance of all models.
  • LLM Cost and Accessibility: The paper acknowledges that while LLMs can offer performance gains, their computational cost and proprietary nature (e.g., GPT-4) raise questions about their practical feasibility for widespread use in low-resource settings.
  • Nahuatl's Complexity: The difficulty encountered with Nahuatl in ST2 underscores the need for more specialized approaches for highly morphologically complex languages.
  • Future Directions: The authors suggest continued efforts in data collection and creation, exploring more robust LLM adaptation strategies, and developing better evaluation metrics tailored to the unique linguistic properties of Indigenous languages. The paper also emphasizes the importance of open science practices, encouraging teams to share their models and data.

Summary updated

BibTeX
@inproceedings{gibert2025findings,
  author = {Ona De Gibert and Robert Pugh and Ali Marashian and Raul Vazquez and Abteen Ebrahimi and Pavel Denisov and Enora Rice and Edward Gow-Smith and Juan Prieto and Melissa Robles and Rubén Manrique and Oscar Moreno and Angel Lino and Rolando Coto-Solano and Aldo Alvarez and Marvin Agüero-Torales and John E. Ortega and Luis Chiruzzo and Arturo Oncevay and Shruti Rijhwani and Katharina von der Wense and Manuel Mager},
  title = {{Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure