TL;DR
This paper tackles the challenging task of morphological segmentation for Seneca, a critically endangered and morphologically rich Native American language. By exploring different training and evaluation strategies, the authors demonstrate the effectiveness of neural sequence-to-sequence models, even with limited data, for this low-resource setting.
What Problem Does This Paper Address?
Morphological segmentation, the process of breaking down words into their smallest meaningful units (morphemes), is crucial for many Natural Language Processing (NLP) tasks like machine translation and speech recognition. However, this task is particularly difficult for morphologically rich languages, where words can be very long and complex, often expressing the meaning of entire sentences. Seneca, a language with fewer than 50 first-language speakers, exemplifies this challenge. Existing research on morphological segmentation has largely focused on high-resource languages, leaving a significant gap in understanding how to effectively segment languages with limited data and complex morphology. This paper aims to bridge this gap by developing and evaluating models for Seneca, contributing to both NLP research and the language's documentation and revitalization efforts.
Methods
The core of this research utilizes neural sequence-to-sequence (seq2seq) models, specifically recurrent neural networks (RNNs) with attention mechanisms, implemented using the OpenNMT toolkit. The authors explore several training strategies:
- In-domain training: Models are trained and evaluated solely on Seneca data.
- Cross-domain training: Models leverage data from related languages or different sources of Seneca data to improve performance. Techniques include:
- Self-training: Using a model to segment unlabeled data and then incorporating that data into the training set.
- Multi-task learning: Training the model to perform morphological segmentation while simultaneously learning to copy input sequences, helping with accurate segmentation of full words.
- Transfer learning: Pre-training a model on data from other languages or domains and then fine-tuning it on Seneca.
- Fine-tuning: Further training a pre-trained model on Seneca data.
- Cross-linguistic training: Utilizing data from other polysynthetic languages (specifically four Yuto-Aztecan languages) to improve Seneca segmentation.
Two evaluation designs are employed:
- Standard evaluation: Using an in-domain development set for model selection.
- Development domain evaluation: Using the development set from a different domain (e.g., informal sources) to evaluate models trained on a specific domain (e.g., grammar book), aiming to maximize the use of available data.
Datasets
The primary dataset for Seneca comes from three sources:
- Grammar book: "The Seneca Verb: Labeling the Ancient Voice" by Bardeau (2007), providing morphologically segmented verbs based on grammatical rules.
- Informal sources: Labeled segmentations of verbs and nouns provided by community speakers.
- Bible: A recently digitized Bible translated into Seneca, offering unlabeled data.
For cross-linguistic experiments, data from four Mexican indigenous languages (Mexicanero, Nahuatl, Wixarika, and Yorem Nokki) from the Yuto-Aztecan family were used.
Key Insights and Ideas
- Neural seq2seq models are effective for low-resource, complex languages: Despite the challenges posed by Seneca's morphology and limited data, neural models achieve surprisingly high performance.
- Development domain evaluation is a viable strategy for low-resource settings: Using data from a related domain for evaluation, rather than strictly in-domain data, can be effective and allows for better utilization of available resources, especially when in-domain development sets are scarce.
- Multi-task learning shows promise: The addition of a task to simply copy input sequences significantly aids the model in learning accurate morphological segmentation.
- Cross-linguistic and cross-domain approaches offer benefits: Leveraging data from other languages or domains can improve segmentation performance, especially when in-domain data is scarce.
- Morfessor struggles with Seneca: While unsupervised Morfessor is a common baseline, it performs significantly worse than the neural models for Seneca, likely due to the language's complex morphology and the frequency of unique morphemes.
Key Results
- High performance on grammar book data: With the development set evaluation, the "less naive baseline" achieved over 91% accuracy and 95% F1 score. Even with the development domain evaluation, neural models consistently outperformed Morfessor.
- Strong performance on informal sources: While performance was lower than for the grammar book data, neural models still achieved over 73% accuracy and 86% F1 score using cross-linguistic multi-task learning with development domain evaluation.
- Model stability: The selected model settings demonstrated stable performance across different random splits of the data, indicating robustness.
- Development domain evaluation shows slight improvement: Evaluating with a development domain, compared to an in-domain development set, led to a slight improvement in average scores for most metrics.
Limitations and Future Work
- Focus on surface segmentation: The study primarily addresses surface segmentation, where the segmented morphemes retain their original orthography. Canonical segmentation, which allows for orthographic changes, was not explored due to the lack of labeled canonical data for Seneca.
- Limited data size: While the study explores low-resource settings, the overall size of the available labeled data, even with the Bible dataset, is still relatively small.
- Future work: The authors plan to refine their models and explore the utility of morphological segmentation for improving language modeling in Automatic Speech Recognition (ASR) systems. They also aim to support transcription of both archival and new recordings for language revitalization projects.
Summary updated