TL;DR
This paper presents a finite-state morphological analyzer for Gitksan, an endangered Indigenous language. The analyzer leverages existing resources and extends them with dialectal relaxation rules and a neural guesser to improve coverage, particularly for less-documented dialects, while maintaining high precision.
What Problem Does This Paper Address?
Documenting and preserving endangered languages is a significant challenge, often hampered by a lack of computational tools for linguistic analysis. Morphological analyzers are crucial for speeding up data analysis, aiding language learning, and expanding lexical resources. Gitksan, an Indigenous language of Western Canada, faces these challenges. While existing dictionaries are limited, this paper aims to build a flexible morphological analyzer that can handle the language's complex morphology, including its strict VSO word order, fusional morphology, and extensive use of clitics. A key challenge is the existence of two main dialects (Eastern and Western) with orthographic variations, which complicates analysis.
Methods
The core of the analyzer is built using finite-state transducer (FST) technology, specifically employing the lexc and xfst tools compiled with foma. This rule-based approach relies on a lexicon of stems and a morphotactic description to generate analyses. To address the limitations of the FST, two extensions were developed:
- Dialectal Relaxation Rules: These rules model orthographic variations between the Eastern and Western dialects, allowing the analyzer to better handle forms from the Western dialect.
- Neural Guesser: A data-driven neural component is added to analyze wordforms that the FST cannot parse. Two neural architectures were explored: a Hard-Attentional model and a Transformer model. This component generates potential analyses for out-of-vocabulary words, which can then be used to expand the FST's lexicon.
The analyzer adopts a maximal-segmentation approach, adding morphemes individually and deriving interactions through transformational rules.
Datasets
- Lexical Resource: A wordlist of approximately 1250 tokens, primarily from the Eastern dialect (Git-an'maaxs), originally documented by Hindle and Rigsby (1973) and updated.
- Corpus: An 18,000-token interlinear-annotated text collection comprising oral narratives from three speakers representing Eastern and Western dialects. This corpus is used for evaluation.
- Evaluation Datasets:
- Two 2000-token datasets from the multidialectal corpus (one Eastern, one Western).
- Smaller datasets for evaluating the Dialectal FST: 150 tokens Eastern and 250 tokens Western.
- Top-100 most frequent word/analysis pairs from the larger datasets.
Key Insights and Ideas
- FSTs are powerful for rule-governed morphology: The paper demonstrates that FSTs are well-suited for capturing Gitksan's complex morphology, including stem-external inflection, cliticization, and morphophonological alternations.
- Dialectal variation is a major hurdle: The significant differences between Eastern and Western dialects, even in orthography, necessitate specific strategies like dialectal relaxation rules to achieve broader coverage.
- Neural models can augment FSTs for coverage: While FSTs offer precision, neural models can effectively bridge the gap for unknown words, significantly increasing overall coverage, albeit with a trade-off in precision.
- Lexical gaps are a primary source of errors: A substantial portion of unanalyzed or incorrectly analyzed forms are due to novel lemmas not present in the lexicon or variants not covered by existing rules. This highlights the importance of ongoing lexical documentation.
- The analyzer as a documentary tool: The FST analyzer is presented not just as a linguistic tool but also as a valuable component of a documentary workflow, helping to identify novel lexical items and grammatical rules from texts.
Key Results
- FST Coverage: The base FST achieved coverage of 71% for the Eastern dialect and 53% for the Western dialect on a manually annotated test set.
- Dialectal FST Improvement: The Dialectal FST (v3) increased Western dialect coverage by approximately 9% (types) and 6% (tokens) without affecting Eastern dialect coverage.
- Precision: The FST analyzers maintained high precision (95-100%) when they were able to provide a parse.
- FST + Neural Performance: The addition of neural components (v4) significantly increased coverage (mean HA: +21%, Fairseq: +17%) but decreased precision (mean -15%).
- Error Analysis: New lemmas and lexical variants accounted for a significant portion of errors (around 18% in small datasets). Specific dialectal realizations of functional items and issues with morphotactic rules (e.g., number agreement) were also identified.
Limitations and Future Work
- Lexicon Size: The primary limitation is the relatively small size of the initial lexicon, which leads to a significant number of unanalyzed forms.
- Neural Precision: The neural component, while improving coverage, reduces precision, requiring careful consideration of how to best integrate its outputs.
- Future Work:
- Expand the FST lexicon by incorporating novel lemmas identified by the analyzer.
- Further refine dialectal relaxation rules.
- Explore alternative neural architectures or training strategies to improve precision.
- Use the neural system as a feedback mechanism for human annotators to identify new data for the FST.
- Develop the analyzer into a pedagogical tool for language learners.
Summary updated