Developing finite-state language technology for Maya

Robert Pugh, Francis Tyers, Quetzil CastañedaACL 2023

TL;DR

This paper introduces a suite of finite-state language technologies for Maya, a Mayan language spoken in Mexico. The core is a computational model of Maya morphology and phonology, resulting in a morphological analyzer and a spell-checker designed to aid both language learners and general processing of written Maya.

What Problem Does This Paper Address?

Developing language technologies for "mainstream" languages like English and Spanish is relatively straightforward due to abundant data and established tools. However, for the vast majority of the world's languages, especially those considered "low-resource" or endangered, such technologies are scarce. This lack of digital tools creates significant barriers for speakers, potentially contributing to language shift towards dominant languages. Specifically for Maya, a language with a rich and complex morphology, existing digital tools are insufficient, hindering its use in education and digital communication.

Methods

The authors leverage finite-state transducers (FSTs), a well-established computational linguistic technique, to model Maya's morphology and phonology. FSTs are particularly attractive for low-resource languages as they require less data than statistical approaches. The core of their work is a morphological analyzer built using the Apertium platform. This platform allows for modular development, separating models for morphotactics (word structure), phonological processes, and analysis disambiguation.

Key methodological aspects include:

  • Morphotactics Modeling: Encoding lexical information about verb roots (transitive, intransitive, positional) and their derivational processes.
  • Phonological Process Modeling: Using the twol (two-level morphology) formalism to handle processes like vowel harmony and common contractions.
  • Analysis Disambiguation: Employing Constraint Grammar to resolve ambiguities arising from Maya's complex morphology, using surrounding context to determine the correct analysis.
  • Spell-Checking: Integrating the morphological analyzer with an FST to create a spell-checker that can identify and suggest corrections for incorrectly spelled words based on edit distance and morphological analysis.

Datasets

The primary dataset used for development is a small corpus of pedagogical texts used in OSEA Maya language classrooms. This corpus includes lists of sentences and dialogues, supplemented by four short stories from Bolles and Bolles (2001). For the spell-checker, the authors mention using an FST that transduces words within a fixed edit-distance to the words in their model, implying the morphological analyzer's lexicon serves as the basis for this.

Key Insights and Ideas

  • Finite-State Transducers for Rich Morphology: The paper demonstrates the effectiveness of FSTs in handling the complex derivational and inflectional morphology of Maya, a language with intricate verb structures.
  • Bridging Normative and Descriptive Models: A key design goal is to support both a "normative" model for pedagogical use (encouraging standardized writing) and a "descriptive" model for broader NLP tasks (accommodating natural variation). This is achieved through features like "Direction" flags and spellrelax files in Apertium.
  • Pedagogical Value of NLP Tools: The work highlights how language technologies can serve as valuable aids for language learners and educators, providing lexical information, aiding reading, and supporting consistent writing practices.
  • Leveraging Existing Platforms: The use of the Apertium platform is crucial, allowing for efficient development and extensibility to other applications like spell-checking and potentially machine translation.

Key Results

  • The morphological analyzer achieves approximately 96% coverage on tokens and 85% on types within their modest corpus.
  • The spell-checker prototype, integrated with LibreOffice Writer, demonstrates the practical application of the morphological analyzer for identifying and correcting misspellings, even for inflected forms not explicitly in a lexicon.
  • The system successfully handles various orthographic conventions, including differences in the representation of [j] vs. [h], tone marking, and colonial-style spellings.

Limitations and Future Work

  • Corpus Size: The authors acknowledge that their corpus is modest and plan to expand it by normalizing the orthography of additional texts.
  • Coverage Gaps: Currently, missed words are primarily interjections or foreign loans, and hapax legomena.
  • Spell-Checker Improvement: Future work includes adding a better-informed error model for the spell-checker, incorporating common misspellings, and weighting it using proofread text.
  • Broader Integration: Plans include exploring browser extensions to aid users in understanding Maya texts and extending spell-checker support to other word processors.
  • Extensibility: The Apertium platform's modularity allows for straightforward extension to other applications beyond morphological analysis and spell-checking.

Summary updated

BibTeX
@inproceedings{pugh2023developing,
  author = {Robert Pugh and Francis Tyers and Quetzil Castañeda},
  title = {{Developing finite-state language technology for Maya}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure