Abstract: Language documentation projects often involve the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. However, there are few existing resources providing large amounts of standardized, easily accessible IGT data, limiting their applicability to linguistic research, and making it difficult to use such data in NLP modeling. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. We normalize much of our data to follow a standard set of labels across languages. Furthermore, we explore the task of automatically generating IGT in order to aid documentation projects. As many languages lack sufficient monolingual data, we pretrain a large multilingual model on our corpus. We demonstrate the utility of this model by finetuning it on monolingual corpora, outperforming SOTA models by up to 6.6%. Our pretrained model and dataset are available on Hugging Face.
The Challenge of Interlinear Glossed Text
Linguists documenting endangered languages rely on Interlinear Glossed Text (IGT) to capture the relationship between speech and grammar. This format typically follows a three-tiered structure: the original transcription, a morpheme-by-morpheme breakdown, and a free translation. While IGT is the gold standard for documentation, its creation is incredibly labor-intensive, requiring deep linguistic expertise for every sentence.
Historically, the NLP community has struggled to automate this process due to two primary hurdles: data scarcity and a lack of standardization. Most endangered languages have very little digitized IGT data, and existing datasets often use inconsistent glossing conventions (e.g., using 1SG, 1.SG, or SING for the same feature). GlossLM addresses these "data silos" by aggregating scattered datasets into a single, normalized corpus.
Methods: Data Engineering and Byte-Level Modeling
To build a robust system, researchers took a two-pronged approach focusing on data normalization and model pretraining.
Data Normalization
Diverse glossing labels from various sources were mapped to the UniMorph schema, a standardized framework for morphological features. Analysis revealed that the top 200 most common glosses account for over 80% of the data. This allowed for the standardization of a significant portion of the corpus, despite the "long tail" of unique linguistic features found across 1,800 languages.
The GlossLM Model
Rather than building language-specific models, the authors developed a single massively multilingual model using ByT5. ByT5 is a version of the T5 transformer that operates on raw bytes rather than subword tokens. This is a critical design choice for endangered languages, which often use unique scripts or complex word structures that standard tokenizers would fail to process correctly. Glossing is framed as a text-to-text problem: the model is prompted with the transcription and translation and tasked with generating the corresponding gloss line.
The GlossLM Corpus
The GlossLM Corpus was compiled from six primary sources, including ODIN (scraped from linguistic PDFs), the SIGMORPHON 2023 Shared Task, and IMTVault. In total, the corpus contains 450,000 instances (250,000 unique sentences) covering 1,800 languages. This represents a massive leap in scale over previous collections, providing a foundation for cross-lingual transfer learning.
Key Results and Insights
Performance and Low-Resource Gains
GlossLM outperformed previous state-of-the-art models (such as the TÜ-CL system) on five out of seven evaluation languages. The most significant gains were observed in low-resource settings. For example, on Gitksan (a language with minimal training data), GlossLM produced well-formatted output where baseline models failed entirely.
Format Transfer and Normalization
A striking finding was that the model benefits from pretraining even on languages linguistically unrelated to the target. This suggests the model learns the structure and logic of IGT—such as how to align morphemes with labels—rather than just vocabulary. Furthermore, standardizing labels to UniMorph improved performance significantly for languages with moderate data, such as Lezgi, which saw a 3.0% accuracy boost.
Limitations and Future Directions
Despite its success, the dataset remains "top-heavy," with Arapaho accounting for roughly 20% of the corpus. This imbalance can bias the model toward specific grammatical structures. Additionally, some gloss labels remain inherently ambiguous, requiring more context-aware normalization tools in the future. Finally, current metrics like "morpheme accuracy" are strict; future work may involve developing more "linguistically-aware" evaluation metrics that account for minor punctuation or formatting differences.
By releasing both the corpus and the model on Hugging Face, the authors have provided a powerful new toolkit for linguists racing to document the world’s disappearing languages.
Summary updated