Abstract
A key aspect of language documentation is the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. Prior work has explored methods to automatically generate IGT in order to reduce the time cost of language analysis. However, many languages (particularly those requiring preservation) lack sufficient IGT data to train effective models, and crosslingual transfer has been proposed as a method to overcome this limitation. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. Then, we pretrain a large multilingual model on a portion of this corpus, and further finetune it to specific languages. Our model is competitive with state-of-the-art methods for segmented data and large monolingual datasets. Meanwhile, our model outperforms SOTA models on unsegmented text and small corpora by up to 6.6% morpheme accuracy, demonstrating the effectiveness of crosslingual transfer for low-resource languages.
The Challenge of Linguistic Documentation
In the field of linguistics, Interlinear Glossed Text (IGT) serves as the gold standard for documenting and analyzing languages. A standard IGT entry typically consists of three distinct layers: the original transcription, a morpheme-by-morpheme breakdown (the "gloss"), and a free translation into a high-resource language like English. This granular annotation is essential for understanding the morphosyntactic structures of the world's languages, yet the process is incredibly labor-intensive, requiring significant expert knowledge. This manual bottleneck is a primary obstacle in documenting the world’s 3,000+ endangered languages.
While Natural Language Processing (NLP) has attempted to automate this task, two major hurdles have persisted: extreme data scarcity for most languages and the limitations of monolingual models. When a language has only a few hundred lines of documented text, traditional models struggle to learn the complex morphological patterns required for accurate glossing.
The GlossLM Approach
To address these challenges, GlossLM introduces a "massively multilingual" approach. The researchers aggregated six major IGT sources—including ODIN, SIGMORPHON, and IMTVault—into the GlossLM Corpus, the largest standardized collection of its kind with 450k instances across 1,800 languages. A critical component of this work was data normalization: the team mapped over 80% of diverse, idiosyncratic grammatical labels to the standardized UniMorph schema, allowing the model to recognize shared grammatical concepts (like "plurality") across different language families.
Technically, the system utilizes ByT5, a byte-level transformer model. Unlike standard subword tokenization, which often fails on rare languages or complex scripts, ByT5 operates on raw UTF-8 bytes. This enables the model to process any character without a fixed vocabulary. The model underwent "continual pretraining" on the massive IGT corpus using a text-to-text prompt format, where it receives a transcription and translation to generate the corresponding gloss line. Finally, the model was finetuned on specific target languages to evaluate the efficacy of crosslingual transfer.
Experimental Results
GlossLM achieved new state-of-the-art (SOTA) results on 5 out of 7 evaluation languages from the SIGMORPHON 2023 Shared Task. The model particularly excelled in the "closed track"—the difficult task of generating glosses from unsegmented text. This is a more realistic scenario for field linguistics, where the exact boundaries between morphemes are often unknown to the researcher.
Key findings include:
- Performance Gains: In the unsegmented track, GlossLM outperformed previous best models by up to 6.6% in morpheme accuracy.
- Low-Resource Robustness: On languages with minimal data, such as Nyangbo (nyb), the pretrained GlossLM significantly outperformed models trained from scratch, proving that pretraining on 1,800 languages helps the model learn the fundamental logic of linguistic annotation.
- Zero-Shot Potential: The model demonstrated a surprising ability to produce well-formatted IGT even for languages with fewer than 100 training examples, where baseline models typically failed entirely.
Discussion and Future Work
Despite these advancements, limitations remain. The GlossLM dataset is "long-tailed," with a few languages like Arapaho accounting for a large portion of the data. Furthermore, while the model is highly accurate with grammatical tags (e.g., "PAST" or "3PL"), it occasionally struggles with lexical glosses, sometimes hallucinating meanings from the English translation that are not present in the source text. Future work will focus on improving robustness to varying researcher styles and enhancing the model's ability to handle lexical roots accurately. By releasing the model and dataset on Hugging Face, the authors provide a powerful new resource for the global language documentation community.
Summary updated