TL;DR
This paper introduces the SIGMORPHON 2023 Shared Task on Interlinear Glossing, a novel initiative to automate the creation of interlinear glosses, a crucial but labor-intensive component of language documentation. The task, featuring two tracks (closed and open resource), saw participants develop systems that significantly outperformed a baseline, with the winning team TÜ-CL achieving substantial improvements.
What Problem Does This Paper Address?
Language documentation is vital for preservation and revitalization, but it's a time-consuming process. A key step is creating interlinear glosses, which break down words into their constituent morphemes and provide their meaning or grammatical function. While interlinear glossed text is a common format, automating its creation has been relatively underexplored in Natural Language Processing (NLP). This shared task aims to bridge this gap by fostering research into automated interlinear glossing, addressing the challenge of disambiguating morphemes and accurately translating word stems, especially for resource-scarce languages.
Methods
The shared task featured two tracks:
- Closed Track: Systems only had access to the orthographic transcription and translation of the target utterance. This forces models to infer morpheme boundaries and identities without explicit segmentation information.
- Open Track: Participants could utilize additional resources like morphological segmentations, dictionaries, and raw text, simulating more realistic language documentation scenarios.
Several teams explored various techniques, including:
- Transformer architectures: RoBERTa and ByT5 were used as baselines and for fine-tuning.
- Hard attention models: Employed by the winning team TÜ-CL to induce shallow morphological segmentation.
- Hybrid CRF-neural systems: Combining traditional and neural approaches.
- Data augmentation: Artificially generating glossing data.
- Cross-lingual transfer: Leveraging knowledge from other languages.
- Translation-based methods: Using sentence translations to aid in glossing word stems.
Datasets
The shared task datasets comprised data from six typologically diverse languages: Arapaho, Gitksan, Lezgi, Natügu, Tsez, and Uspanteko. These datasets were derived from existing corpora and converted into a unified format adhering to Leipzig glossing conventions. The data was split into non-overlapping training, development, and test sets. For most languages, an 80-10-10 split was used, while for Gitksan and Arapaho, complete texts were allocated to development and testing to leverage existing divisions.
Key Insights and Ideas
- Automating glossing is feasible and impactful: The shared task demonstrated that significant improvements in glossing accuracy are achievable over baseline systems, even in resource-scarce settings.
- The value of morphological segmentation: Providing morphological segmentations (as in the open track) dramatically improves performance, highlighting the difficulty of inferring boundaries from scratch.
- Translations are useful, but not always essential: The winning system TÜ-CL achieved top results without using translations, suggesting that while helpful, they might not be the sole path to high performance, especially with advanced models.
- Low-resource languages remain challenging: Despite advancements, languages with very small training sets (like Gitksan) consistently showed the lowest performance, underscoring the ongoing need for low-resource NLP techniques.
- Morpheme-level accuracy is harder than word-level: Accuracies for morpheme-level glossing were generally lower than word-level, likely due to the cascading errors from incorrect morpheme boundary identification.
Key Results
- Winning Performance: The TÜ-CL team achieved the best results, with a 23.99%-point improvement over the baseline in the closed track and a 17.42%-point improvement in the open track.
- Track 1 (Closed): TÜ-CL achieved a micro-average word-level accuracy of 71.30%. COATES also outperformed the baseline.
- Track 2 (Open): TÜ-CL again led with a micro-average word-level accuracy of 71.30% (note: the paper states 71.30% for both tracks for TÜ-CL, which might be a typo and should reflect the 17.42% improvement for the open track). SIGMOREFUN and LISNTEAM also showed strong performance in specific languages.
- Training Data Size Matters: Performance generally correlated with the size of the training data, with Arapaho (largest dataset) showing high performance. Gitksan (smallest dataset) consistently performed the worst.
- OOV and TTR Impact: The impact of Out-of-Vocabulary (OOV) rate and Type-Token-Ratio (TTR) on performance was less clear-cut than training data size, with some exceptions and potential confounding factors.
Limitations and Future Work
- Limitations:
- The baseline system struggled with out-of-vocabulary words and did not perform segmentation in the closed track.
- The impact of OOV and TTR was not consistently clear across all languages.
- Pretrained models like ByT5 did not perform as well as expected on this task compared to other morphology tasks.
- Future Work:
- Further investigation into why ByT5 underperformed is needed.
- Exploring the use of large pretrained English models to enhance translation-based glossing.
- Leveraging multi-task learning with existing morphological resources (like UniMorph).
- Investigating cross-lingual approaches and utilizing noisy but multilingual datasets like ODIN.
Summary updated