JHU’s Submission to the AmericasNLP 2025 Shared Task on the Creation of Educational Materials for Indigenous Languages

Tom Lupicki, Lavanya Shankar, Kaavya Chaparala, David YarowskyNAACL 2025

TL;DR

This paper details JHU's successful submission to the AmericasNLP 2025 shared task, which focuses on automatically generating educational materials for endangered Indigenous languages. By ensembling multiple large language models (LLMs) and incorporating linguistic insights, they achieved top performance on Maya and Guaraní, demonstrating a promising path for preserving and teaching these languages.

What Problem Does This Paper Address?

Many Indigenous languages in the Americas are endangered, with dwindling speaker populations and a lack of sufficient data to leverage modern Natural Language Processing (NLP) advancements. This poses a significant challenge for creating educational resources, which often rely on the expertise of a small number of fluent speakers. The AmericasNLP 2025 shared task aimed to address this by challenging participants to develop systems that can automatically generate grammatical exercises for languages like Bribri, Maya, Guaraní, and Nahuatl. These exercises involve transforming base sentences based on specific grammatical rules (e.g., tense changes, negation), thereby reducing the burden on language experts and making language learning more accessible.

Methods

The JHU team employed a multi-pronged approach, combining several state-of-the-art techniques:

  • LLM Prompting: They experimented with few-shot prompting, providing LLMs with examples of source sentences, grammatical changes, and target sentences. Variations included augmenting prompts with linguistic information, part-of-speech tags, and employing "chain-of-thought" (CoT) reasoning to guide the LLMs' step-by-step problem-solving.
  • Ensembling: A core strategy was ensembling multiple LLM outputs. They used majority voting at different granularities (token-level, sentence-level) to combine predictions from up to six different LLM configurations. This approach proved highly effective.
  • Pointer-Generator Network: As a contrastive system, they trained a character-level pointer-generator LSTM, a more traditional sequence-to-sequence model, to compare its performance against LLM-based approaches.
  • Data Augmentation: For Bribri, they created additional training data by extracting verb conjugation tables from a reference grammar, aiming to bolster performance for this specific language.

Datasets

The task provided training, development, and test datasets for four languages: Bribri, Maya, Guaraní, and Nahuatl. The datasets contained base sentences, grammatical modifications, and the corresponding correctly transformed sentences. For Bribri, the authors supplemented the provided data with approximately 1400 additional single-verb training examples derived from a reference grammar.

Key Insights and Ideas

  • Ensembling is Key: The most significant finding is the power of ensembling multiple LLM outputs. Their majority voting ensemble (System 1) significantly outperformed individual LLM systems and was declared a winning system.
  • Prompt Engineering Matters, But Not Always Predictably: While prompt variations (e.g., number of examples, inclusion of POS tags or reference book info) were explored, the impact on performance wasn't always straightforward. This suggests that the underlying LLM capabilities and the quality of the few-shot examples might be more critical than minor prompt tweaks.
  • Nahuatl Remains a Challenge: The team observed consistently poor performance on Nahuatl across all their systems. This highlights the significant linguistic diversity within Nahuatl varieties and the difficulty LLMs face in handling such nuances without specialized training or data.
  • Leveraging Linguistic Resources: The inclusion of linguistic information (though not always directly beneficial in their experiments) points to the potential of integrating external knowledge to improve LLM performance on low-resource languages.

Key Results

JHU's System 1, a majority voting ensemble of LLMs, achieved the highest scores for Maya and Guaraní, making it one of the two winning systems for the task. Specifically, their ensemble system provided substantial improvements over previous years' winning systems for these languages, with an additional 10.00 percentage points in accuracy for Maya and 9.06 percentage points for Guaraní. For Bribri, their ensembling strategy also yielded improvements of 2.50, 3.87, and 2.75 percentage points in accuracy for Bribri, Maya, and Guaraní, respectively, compared to their best single LLM submissions.

Limitations and Future Work

The authors acknowledge the poor performance on Nahuatl as a key limitation. They hypothesize this is due to the extreme linguistic diversity within Nahuatl varieties and suggest that future work should focus on more variety-specific approaches, potentially involving sourcing and incorporating grammatical information for Western Sierra Puebla Nahuatl and fine-tuning LLMs on data specific to this variety. They also propose that further analysis could explore whether incorrect outputs for Nahuatl might be valid for other varieties, offering insights into system adaptability. Additionally, they suggest that future work could investigate prompt configurations more granularly to better understand their impact.

Summary updated

BibTeX
@inproceedings{lupicki2025jhus,
  author = {Tom Lupicki and Lavanya Shankar and Kaavya Chaparala and David Yarowsky},
  title = {{JHU’s Submission to the AmericasNLP 2025 Shared Task on the Creation of Educational Materials for Indigenous Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure