TL;DR
This paper tackles the challenge of automatically generating interlinear glosses for Otomi, a low-resource indigenous language. By employing Conditional Random Fields (CRFs) with carefully selected linguistic features, they achieve high accuracy, demonstrating the effectiveness of traditional machine learning models over neural networks in such data-scarce scenarios. This work offers a valuable tool for language documentation and can pave the way for developing NLP technologies for Otomi.
What Problem Does This Paper Address?
Interlinear glossing is a fundamental linguistic task that involves segmenting words into their constituent morphemes and assigning each morpheme a grammatical label. This process is crucial for language documentation and provides valuable data for Natural Language Processing (NLP) applications. However, manually creating these glosses is time-consuming and requires linguistic expertise. This is particularly problematic for low-resource languages, like Otomi, which often lack extensive digital documentation and existing NLP tools. The paper aims to automate this glossing process for Otomi, bridging the gap between linguistic analysis and computational linguistics in a challenging low-resource setting.
Methods
The authors explored several sequential labeling models for automatic glossing. Their primary approach involved Conditional Random Fields (CRFs), specifically linear-chain CRFs. They experimented with different feature sets for the CRFs, including information about the current character, its position within the word and sentence, surrounding characters, and importantly, Part-of-Speech (POS) tags. They also investigated the impact of excluding POS tags.
For comparison, they implemented and evaluated two other sequential models:
- Hidden Markov Models (HMMs): A generative baseline model.
- Recurrent Neural Networks (RNNs): Including a vanilla RNN and a bidirectional LSTM (biLSTM).
The glossing process involved a two-step approach: first, segmenting words into morphemes, and second, assigning a gloss label to each morpheme. They used a BIO (Beginning-Inside-Outside) labeling scheme to mark morpheme boundaries.
Datasets
The primary dataset used in this study is a corpus of Otomi of Toluca. This corpus was originally transcribed into a phonetic alphabet and was manually digitized and orthographically normalized. It consists of narrations and dialogues, totaling 1769 sentences and 8550 tokens. Additionally, 81 extra short sentences containing challenging linguistic phenomena were included.
For evaluation, the dataset was split into training and testing sets. For the CRF models, a 3-fold cross-validation was used due to the small dataset size. For the HMM and RNN models, a hold-out evaluation was performed, preserving the same proportion between training and test datasets.
Key Insights and Ideas
- CRFs Outperform Neural Networks in Low-Resource Settings: Surprisingly, traditional CRF models achieved significantly better accuracy than more complex neural network architectures (vanilla RNN and biLSTM) for this low-resource language. This suggests that for tasks with limited data, carefully engineered features for CRFs can be more effective than the data-hungry nature of deep learning models.
- Feature Engineering is Crucial: The performance of the CRF models was highly dependent on the features used. Including contextual information (previous and next characters, POS tags) significantly boosted accuracy.
- POS Tags May Not Be Essential: Excluding POS tags from the CRF features (CRFPOSLess setting) resulted in only a minor drop in accuracy compared to the CRFlinear setting, which included all features. This is a significant finding, as POS tagging itself can be a challenging task for low-resource languages.
- Linguistic Structure Matters: The paper highlights how Otomi's complex morphological system, with prefixes, suffixes, and stem alternations, necessitates contextual information for accurate glossing.
- Productive Morphemes are Easier to Gloss: Linguistically, frequently occurring and systematic morphemes (like "stem" or "3.CPL") were easier for the models to predict, while less frequent or less systematic elements (like "PRT") showed lower performance.
Key Results
The CRF-based models consistently outperformed the HMM and RNN models.
- The CRFlinear model, using all available features, achieved the highest average accuracy of 96.2%.
- The CRFPOSLess model, excluding POS tags, achieved a very competitive accuracy of 94.8%.
- The CRFHMMLike model, using minimal features, achieved 88.0% accuracy.
- The HMM baseline achieved 87.8% accuracy.
- The vanilla RNN achieved 74.1% accuracy.
- The biLSTM model performed the worst with 56.3% accuracy.
Limitations and Future Work
- Limited Corpus Size: The primary limitation is the small size of the Otomi corpus, which is typical for low-resource languages. This restricts the complexity of models that can be trained effectively.
- Translation Not Addressed: The paper focuses only on morphological segmentation and glossing, not on translating the sentence into another language.
- Model Sensitivity: The neural models were observed to be highly sensitive to label frequency, performing better on common labels.
- Future Work: The authors suggest exploring more complex CRF models and investigating how to best leverage linguistic knowledge for feature engineering. They also propose that this work can serve as a starting point for developing other NLP technologies for Otomi, such as morphological segmentation and reinflection systems. The potential for complex models with many parameters to be inefficient in low-resource settings is noted as an interesting research question for the future.
Summary updated