Experiments in Multi-Variant Natural Language Processing for Nahuatl

Robert Pugh, Francis TyersNAACL 2024

TL;DR

This paper tackles the challenge of building Natural Language Processing (NLP) tools for Nahuatl, a language with many variants but no single standard. By experimenting with different training strategies on two Nahuatl dialects, they demonstrate that jointly training models on multiple variants significantly improves performance, even on unseen dialects, highlighting the importance of diverse data for low-resource languages.

What Problem Does This Paper Address?

Developing robust NLP tools for languages with significant internal variation, especially those lacking a standardized dialect, is a major hurdle. This "dialect gap" leads to performance degradation when models trained on one variety are applied to another. This is particularly true for languages like Nahuatl, an indigenous Mexican language with around 30 recognized varieties, where historical texts often don't reflect contemporary usage, and there's no single dominant standard. The authors aim to explore how to build NLP systems that can handle this multi-variant nature, making technology more accessible to speakers of diverse Nahuatl dialects.

Methods

The core of this research involves training and evaluating neural network models for dependency parsing using the MaChAmp toolkit. They fine-tune pre-trained multilingual BERT embeddings. Their experimental setup includes:

  • Monolingual Training: Training a model on data from a single Nahuatl variant.
  • Cross-Variant Transfer (Zero-Shot): Training a model on one variant and evaluating its performance on another variant without any further training on the target variant's data.
  • Joint Training: Concatenating the training data from multiple Nahuatl variants to train a single model.
  • Volume-Adjusted Joint Training: A variation of joint training where only half of the data from each variant is used, to control for the effect of increased data volume.
  • Auxiliary Task Training (Masked Language Modeling): Incorporating an additional MLM task using unlabeled Nahuatl text to potentially improve the model's understanding of the language.

Datasets

The primary datasets used are:

  • Highland Puebla Nahuatl (azz): Approximately 10,000 tokens from a Universal Dependencies (UD) treebank.
  • Western Sierra Puebla Nahuatl (nhi): Approximately 10,000 tokens from a UD treebank.
  • Axolotl Corpus: A large, unlabeled parallel Nahuatl-Spanish corpus (over 10,000 sentences) used for auxiliary task training.
  • Central Puebla Nahuatl (ncx) - "Casanova stories": A small (approx. 2,000 tokens) annotated dataset of texts from a third, unseen variant, used for evaluating cross-variant performance.

The UD treebanks for azz and nhi were split into training and evaluation sets, with 10-fold cross-validation used for most experiments.

Key Insights and Ideas

  • Joint training is key: Combining data from different Nahuatl variants into a single training set significantly boosts performance, even outperforming models trained solely on one variant when evaluated on that same variant. This suggests that models can learn to generalize across dialects.
  • Dialectal similarity matters: Even superficial alterations to training data to mimic dialectal features (like isoglosses) can improve performance on unseen variants, indicating that linguistic similarity plays a crucial role.
  • Genre can be a confounder: The performance differences between variants might not solely be due to linguistic divergence but also influenced by the genre of the text (e.g., stories vs. plant descriptions).
  • No single standard is a solvable problem: The research demonstrates that with appropriate multi-variant training strategies, NLP tools can be developed for languages like Nahuatl, despite the absence of a standardized dialect.

Key Results

  • Monolingual vs. Cross-Variant: Performance drops significantly when a model trained on one variant is tested on another (cross-variant). For example, the azz monolingual model achieved ~0.92 Lemma accuracy, while the cross-variant performance dropped to ~0.68.
  • Joint Training Performance: Jointly trained models (trained on both azz and nhi) performed comparably to or better than the best monolingual models on their respective variants. For instance, the joint model achieved ~0.92 Lemma accuracy on azz, similar to the azz monolingual model's ~0.92.
  • Performance on Unseen Variant (ncx): The jointly trained model achieved the best performance on the unseen ncx variant across all tasks, outperforming monolingual models trained on either azz or nhi alone. This is a critical finding, showing generalization capabilities.
  • Auxiliary Task Impact: Adding an auxiliary MLM task using unlabeled data did not yield a significant improvement in performance, with results remaining within the margin of error compared to joint training without the auxiliary task.

Limitations and Future Work

  • Limited Variant Coverage: The study only uses two out of the 30 recognized Nahuatl variants. Future work should focus on collecting annotated corpora for more variants and diverse domains.
  • Genre Influence: The authors acknowledge that genre differences between corpora might influence performance, and suggest further investigation into this aspect.
  • Auxiliary Task Effectiveness: The MLM auxiliary task did not show significant gains, suggesting that other pretraining or auxiliary task strategies might be more beneficial.
  • Spanish Influence: Nahuatl's long contact with Spanish presents an opportunity to explore how Spanish data might be leveraged to improve Nahuatl NLP.
  • Investigating Similarity Metrics: Future research could empirically investigate the role of linguistic and genre similarity using various metrics.

Summary updated

BibTeX
@inproceedings{pugh2024experiments,
  author = {Robert Pugh and Francis Tyers},
  title = {{Experiments in Multi-Variant Natural Language Processing for Nahuatl}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure