TL;DR
This paper introduces the first computational morphological transducer and orthography converter for San Lucas Quiaviní Zapotec (SLQZ), an endangered indigenous language. The developed tools aim to support language maintenance efforts by enabling automated analysis and conversion between different orthographic conventions used for the language.
What Problem Does This Paper Address?
Many indigenous languages, like Western Tlacolula Valley Zapotec (specifically San Lucas Quiaviní Zapotec, or SLQZ), face endangerment due to shifting language use towards dominant languages. To support the maintenance and revitalization of these languages, computational tools are crucial. However, the development of such tools, particularly morphological transducers (which analyze word structures), has lagged for many under-resourced languages. Furthermore, SLQZ, like many languages, has multiple orthographic conventions (phonemic vs. simplified practical), creating a barrier for consistent digital processing and inter-orthographic communication. This paper addresses the need for a robust morphological transducer and an orthography converter for SLQZ, paving the way for broader language technology applications.
Methods
The core of this work is the development of a morphological transducer implemented using the two-level approach and the HFST (Finite-State Transducer) toolkit. This approach involves creating separate finite-state transducers (FSTs) for lexicon and morphotactics, and morphophonology, which are then intersected to form a single FST with both an analysis and a form side.
Key methodological aspects include:
- Lexicon Design: The lexicon was built using
lexd, a tool that allows for flexible lexicon management, including handling of different word categories (proper nouns, verbs, etc.) and their associated morphological patterns. - Morphotactics and Morphophonology: These were implemented using hand-written patterns, lexicons, and rules. The
lexdcompiler was used for morphotactics, andtwol(part of HFST) for morphophonology. - Orthography Conversion: A novel approach was used where two transducers (one for each orthography) are intersected along their common analysis side. This allows for conversion between the simplified and phonemic orthographies.
- Handling Orthographic Variants: A "spellrelax" file was used to compile a series of regular expressions into an FST, enabling the analyser to accept forms with various specified orthographic variants (e.g., different apostrophe usage).
Datasets
The primary dataset used for evaluation consists of various texts in San Lucas Quiaviní Zapotec, including:
- Blal xte Tiu Pamyël (BxTP): Portions of this story were used for both development and testing.
- Felipe H. Lopez poetry: A collection of poems.
- Tlalocan poems & story: Texts with individualized orthographies.
- Niny Bac, Liaza Chaa: Other narrative texts.
- Ticha blog post: A contemporary text.
- Universal Declaration of Human Rights (UDHR): An older version of the phonemic orthography.
- New Testament translation in SJGZ: A related Zapotec variety, used to test cross-variety applicability.
The paper notes that BxTP parts 1-2 were used as development data, while the remaining texts were treated as previously unseen.
Key Insights and Ideas
- Feasibility for Under-Resourced Languages: The paper demonstrates that developing sophisticated NLP tools like morphological transducers is feasible even for languages with relatively small corpora, which is a significant finding for endangered language technology.
- Orthography Conversion via FST Intersection: The method of intersecting analysers and generators from different orthographies is a clever way to build converters, leveraging the shared analysis side of FSTs.
- Impact of Coverage on Conversion: The orthography converter's performance is heavily dependent on the transducer's coverage. If a word isn't analyzed, it can't be converted, highlighting the interconnectedness of these NLP components.
- Reciprocity in Open Source Licensing: The authors emphasize the importance of culturally specific licensing models (like the "Guelaguetza clause") for indigenous languages, balancing open-source principles with community values of reciprocity.
Key Results
- Naïve Coverage: The transducer achieved over 90% naïve coverage on development texts (BxTP 1-2). For unseen texts, coverage averaged around two-thirds (67.47% overall). This indicates a solid base but significant room for expansion.
- Orthographic Conversion Accuracy:
- Phonemic to simple conversion showed a Word Error Rate (WER) as low as 1.63% for BxTP 1-2, demonstrating high determinism.
- Simple to phonemic conversion had higher WER (e.g., 11.78% for BxTP 1-2), influenced by the transducer's coverage and the inherent ambiguity of the simplified orthography.
- Cross-Variety Applicability: Applying the SLQZ transducer to the related SJGZ variety resulted in low coverage (around 34%), suggesting that distinct transducers might be necessary for different Zapotec varieties.
Limitations and Future Work
- Incomplete Coverage: The authors acknowledge that the transducer is incomplete, with many opportunities for expansion, particularly in its lexicon and morphological patterns.
- Orthographic Homography: The simplified orthography presents homographs (words spelled the same but with different meanings/analyses), which can lead to incorrect analyses and limit the accuracy of conversion, especially without wider context.
- Cross-Variety Application: The limited success in applying the SLQZ transducer to SJGZ suggests that further work is needed to develop transducers for other Western Tlacolula Valley Zapotec varieties.
Future work will focus on:
- Expanding the lexicon.
- Adding missing morphological patterns.
- Refining morphophonological patterns.
- Developing better strategies for handling orthographic nuances and homography.
- Potentially creating separate transducers for different Zapotec varieties.
Summary updated