TL;DR
This paper introduces the first finite-state morphological analyzer for Shipibo-Konibo, a low-resource indigenous language from Peru. The tool performs morphological segmentation and categorization, significantly aiding research and corpus annotation efforts for this endangered language.
What Problem Does This Paper Address?
Documenting and developing linguistic resources for indigenous languages, especially those with low resource levels, is crucial for their preservation and for enabling further research. Shipibo-Konibo, spoken by around 30,000 people in the Peruvian Amazon, has a complex morphosyntax that presents challenges for computational linguistic tools. While previous work has focused on documentation and manual annotation, a robust, automated morphological analyzer was lacking. This paper aims to fill that gap by providing a freely available tool to facilitate downstream applications and the creation of annotated corpora.
Methods
The authors developed a morphological analyzer for Shipibo-Konibo using the Foma toolkit, a finite-state compiler. The analyzer's design is heavily informed by the comprehensive morphological description of Shipibo-Konibo provided by Valenzuela (2003). The system performs:
- Morphological Segmentation: Breaking down words into their constituent morphemes.
- Morphological Categorization: Assigning grammatical tags to each morpheme.
- Part-of-Speech (POS) Tagging: Identifying the grammatical category of the root and the entire token.
The analyzer categorizes roots and then applies a sequence of derivational processes. The POS tagset used is mapped to the Universal Dependencies v2 tagset. The implementation accounts for the language's predominantly suffixed morphology, including prefixes related to body parts, plural markers, valency-changing suffixes, and various other markers for tense, aspect, mood, and reference.
Datasets
The development and evaluation of the analyzer utilized:
- Annotated Corpus: A dataset of 12,250 tokens and 2,915 types, derived from folk tales and manually annotated.
- Digitalized Thesaurus: A lexical resource containing 6,750 types, provided by the Pontifical Catholic University of Peru.
- Raw Corpora: Various texts from different domains (Bible, elementary school books, kindergarten materials, legal documents, folk tales) were used for evaluation.
Key Insights and Ideas
- Finite-State Systems for Low-Resource Languages: The paper demonstrates the effectiveness of finite-state systems for modeling the complex morphosyntax of low-resource languages, proving robust enough to handle exceptions and variations.
- Importance of Comprehensive Grammars: The analyzer's success is directly tied to the detailed grammatical description by Valenzuela (2003), highlighting the critical role of linguistic documentation.
- Challenges of Historical Texts: The analyzer struggles with older texts (like the Bible translation) due to outdated spellings and grammatical constructions that are no longer standard, underscoring the need for up-to-date linguistic data.
- Ambiguity in Morphological Tagging: Certain morphemes in Shipibo-Konibo have multiple functions, leading to ambiguity that the token-by-token analyzer cannot resolve without further contextual information.
Key Results
The analyzer achieved a type coverage of 94.99% on the training data (annotated corpus + thesaurus). Evaluation on raw corpora showed varying coverage across domains, with folk tales exhibiting the highest coverage (94.38% tokens, 85.42% types) and legal documents the lowest (70.83% tokens, 40.57% types), likely due to specialized vocabulary and complex constructions. An error analysis identified alternative spellings and obsolete constructions as primary sources of unanalyzed words.
Limitations and Future Work
The authors acknowledge several limitations:
- Token-by-Token Processing: The analyzer does not consider broader context, limiting its ability to disambiguate morphemes with multiple functions or resolve rigid constructions (e.g., future tense, modal verbs).
- Lexicon Size: The root lexicon, while substantial, is still limited, particularly for common nouns and other word types, necessitating further expansion.
- Diacritization Issues: The analyzer struggles with variations in diacritization, especially in older texts, due to the recent proposal of standardized diacritization rules.
Future work suggested includes:
- Mapping morphological tags to Universal Dependencies features.
- Developing sentence-level disambiguation.
- Integrating parsing capabilities.
- Expanding the lexicon and incorporating more up-to-date linguistic data.
Summary updated