Py-Elotl: A Python NLP package for the languages of Mexico

Ximena Gutierrez-Vasques, Robert Pugh, Victor Mijangos, Diego Barriga Martínez, Paul Aguilar, Mikel Segura, Paola Innes, Javier Santillan, Cynthia Montaño, Francis TyersNAACL 2025

TL;DR

This paper introduces Py-Elotl, a Python package designed to democratize NLP for indigenous languages of Mexico. It provides essential "upstream" tools like parallel corpora, orthographic normalizers, and morphological analyzers, addressing a critical gap in resources for these under-resourced languages. This work aims to empower researchers and speakers, fostering greater linguistic diversity in the digital realm.

What Problem Does This Paper Address?

While NLP has revolutionized communication for many languages, a significant digital divide persists for thousands of the world's languages, particularly indigenous ones. Mexico, with its rich linguistic diversity, is a prime example, where many indigenous languages lack fundamental NLP tools. This absence hinders linguistic research and the development of applications that could benefit their speakers. The paper addresses this by providing a comprehensive suite of open-source tools for pre-processing text in several Mexican indigenous languages, focusing on the crucial "upstream" tasks that form the bedrock of more advanced NLP applications.

Methods

Py-Elotl employs a combination of established and novel approaches:

  • Parallel Corpus Loading: The package includes loaders for parallel corpora, which are essential for machine translation and comparative linguistic studies. These loaders are designed to be flexible, accommodating various dialectal variations and sources.
  • Orthographic Normalization: This is a key contribution, addressing the challenge of inconsistent spelling in many indigenous languages. Py-Elotl uses finite-state transducers (FSTs) to convert text first to a phonemic representation (IPA) and then to a user-specified orthographic norm. This two-step process allows for standardization across different writing conventions.
  • Morphological Analyzers: The toolkit integrates free and open-source finite-state transducer (FST) morphological analyzers. FSTs are chosen for their effectiveness in low-resource scenarios where large annotated datasets are scarce. The analyzers are unified under a common tagset (Universal Dependencies) for consistency.

Datasets

The paper describes the use of several parallel corpora:

  • Axolotl parallel corpus: A large corpus for Spanish-Nahuatl, compiled from diverse sources and covering multiple dialectal variations.
  • Tsunkua corpus: A corpus for Spanish-Otomi, primarily consisting of translations from historical texts, dialogues, and grammars.
  • Kolo corpus: A smaller corpus for Spanish-Mixtec, built from educational materials, grammars, and short stories, encompassing a wide range of dialectal variations.

The paper also mentions the use of the Universal Dependencies treebank for Western Sierra Puebla Nahuatl for evaluating the orthographic normalizer.

Key Insights and Ideas

  • The "Upstream" Bottleneck: The paper strongly emphasizes that advanced NLP applications (like LLMs) are only possible due to foundational "upstream" resources. For many indigenous languages, these basic building blocks are missing, creating an uneven landscape of language technology.
  • The Importance of Open-Source and Collaboration: Py-Elotl is explicitly designed as an open-source project to encourage collaboration and ensure that technologies are developed in dialogue with indigenous communities, avoiding cultural appropriation.
  • FSTs as a Viable Solution for Low-Resource NLP: The reliance on FSTs for morphological analysis and orthographic normalization highlights their continued relevance and effectiveness in situations where data-driven methods are challenging due to limited annotated data.
  • "Tequilogias" as a Model for Ethical Tech Development: The paper references the concept of "tequiologias" (a blend of "tequio," a form of communal work, and "tecnologia") to advocate for technology development that is deeply integrated with and beneficial to indigenous communities.

Key Results

  • Py-Elotl currently supports four language groups: Nahuatl, Otomi, Mixtec, and Huave.
  • The orthographic normalizer for Nahuatl achieved 98% accuracy on unique words from the Western Sierra Puebla Nahuatl UD treebank.
  • The orthographic normalizer for Otomi achieved 81% accuracy on average when compared to a gold standard.
  • The toolkit provides access to parallel corpora for Spanish-Nahuatl (16K sentences), Spanish-Otomi (5K sentences), and Spanish-Mixtec (2K sentences).
  • Morphological analyzers are available for multiple variants of Nahuatl, Huave, and Otomi.

Limitations and Future Work

  • Limited Language Coverage: The current version of Py-Elotl focuses on a few language groups, and many other indigenous languages and dialects of Mexico remain underrepresented. Expanding coverage is a key future goal.
  • Rule-Based Limitations: The reliance on rule-based FSTs for normalization and analysis may limit their flexibility in handling phenomena like code-switching, ambiguous cases, and non-standard language use, which are part of the linguistic reality for many speakers.
  • Prototype Status of Otomi Analyzer: The Otomi morphological analyzer is noted as being in a "prototype" status, with its coverage and performance not yet fully published.
  • Future Work: The authors aim to expand the toolkit to support more languages and features, encouraging further development of open and collaborative technologies that are aligned with the needs and contexts of the speakers. They also suggest that the FST framework can be extended to create more sophisticated NLP tools.

Summary updated

BibTeX
@inproceedings{gutierrezvasques2025py,
  author = {Ximena Gutierrez-Vasques and Robert Pugh and Victor Mijangos and Diego Barriga Martínez and Paul Aguilar and Mikel Segura and Paola Innes and Javier Santillan and Cynthia Montaño and Francis Tyers},
  title = {{Py-Elotl: A Python NLP package for the languages of Mexico}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure