GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text

Michael Ginn, Lindia Tjuatja, Taiqi He, Enora Rice, Graham Neubig, Alexis Palmer, Lori Levin

Abstract: Language documentation projects often involve the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. However, there are few existing resources providing large amounts of standardized, easily accessible IGT data, limiting their applicability to linguistic research, and making it difficult to use such data in NLP modeling. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. We normalize much of our data to follow a standard set of labels across languages. Furthermore, we explore the task of automatically generating IGT in order to aid documentation projects. As many languages lack sufficient monolingual data, we pretrain a large multilingual model on our corpus. We demonstrate the utility of this model by finetuning it on monolingual corpora, outperforming SOTA models by up to 6.6%. Our pretrained model and dataset are available on Hugging Face.

The Challenge of Interlinear Glossed Text

Linguists documenting endangered languages rely on Interlinear Glossed Text (IGT) to capture the relationship between speech and grammar. This format typically follows a three-tiered structure: the original transcription, a morpheme-by-morpheme breakdown, and a free translation. While IGT is the gold standard for documentation, its creation is incredibly labor-intensive, requiring deep linguistic expertise for every sentence.

An example of Interlinear Glossed Text (IGT) for the Arapaho language. The format includes the original sentence, a morpheme-by-morpheme segmentation, and standardized gloss labels.

Historically, the NLP community has struggled to automate this process due to two primary hurdles: data scarcity and a lack of standardization. Most endangered languages have very little digitized IGT data, and existing datasets often use inconsistent glossing conventions (e.g., using 1SG, 1.SG, or SING for the same feature). GlossLM addresses these "data silos" by aggregating scattered datasets into a single, normalized corpus.

Methods: Data Engineering and Byte-Level Modeling

To build a robust system, researchers took a two-pronged approach focusing on data normalization and model pretraining.

Data Normalization

Diverse glossing labels from various sources were mapped to the UniMorph schema, a standardized framework for morphological features. Analysis revealed that the top 200 most common glosses account for over 80% of the data. This allowed for the standardization of a significant portion of the corpus, despite the "long tail" of unique linguistic features found across 1,800 languages.

The GlossLM Model

Rather than building language-specific models, the authors developed a single massively multilingual model using ByT5. ByT5 is a version of the T5 transformer that operates on raw bytes rather than subword tokens. This is a critical design choice for endangered languages, which often use unique scripts or complex word structures that standard tokenizers would fail to process correctly. Glossing is framed as a text-to-text problem: the model is prompted with the transcription and translation and tasked with generating the corresponding gloss line.

The GlossLM Corpus

The GlossLM Corpus was compiled from six primary sources, including ODIN (scraped from linguistic PDFs), the SIGMORPHON 2023 Shared Task, and IMTVault. In total, the corpus contains 450,000 instances (250,000 unique sentences) covering 1,800 languages. This represents a massive leap in scale over previous collections, providing a foundation for cross-lingual transfer learning.

Key Results and Insights

Performance and Low-Resource Gains

GlossLM outperformed previous state-of-the-art models (such as the TÜ-CL system) on five out of seven evaluation languages. The most significant gains were observed in low-resource settings. For example, on Gitksan (a language with minimal training data), GlossLM produced well-formatted output where baseline models failed entirely.

Morpheme accuracy as a function of training data size. GlossLM (blue) significantly outperforms the ByT5 baseline (grey), particularly in extremely low-resource scenarios like Gitksan (git).

Format Transfer and Normalization

A striking finding was that the model benefits from pretraining even on languages linguistically unrelated to the target. This suggests the model learns the structure and logic of IGT—such as how to align morphemes with labels—rather than just vocabulary. Furthermore, standardizing labels to UniMorph improved performance significantly for languages with moderate data, such as Lezgi, which saw a 3.0% accuracy boost.

Limitations and Future Directions

Despite its success, the dataset remains "top-heavy," with Arapaho accounting for roughly 20% of the corpus. This imbalance can bias the model toward specific grammatical structures. Additionally, some gloss labels remain inherently ambiguous, requiring more context-aware normalization tools in the future. Finally, current metrics like "morpheme accuracy" are strict; future work may involve developing more "linguistically-aware" evaluation metrics that account for minor punctuation or formatting differences.

By releasing both the corpus and the model on Hugging Face, the authors have provided a powerful new toolkit for linguists racing to document the world’s disappearing languages.

Summary updated

Abstract

Language documentation projects often involve the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. However, there are few existing resources providing large amounts of standardized, easily accessible IGT data, limiting their applicability to linguistic research, and making it difficult to use such data in NLP modeling. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. We normalize much of our data to follow a standard set of labels across languages. Furthermore, we explore the task of automatically generating IGT in order to aid documentation projects. As many languages lack sufficient monolingual data, we pretrain a large multilingual model on our corpus. We demonstrate the utility of this model by finetuning it on monolingual corpora, outperforming SOTA models by up to 6.6\%. Our pretrained model and dataset are available on Hugging Face.
BibTeX
@inproceedings{ginn2024glosslm,
  author = {Michael Ginn and Lindia Tjuatja and Taiqi He and Enora Rice and Graham Neubig and Alexis Palmer and Lori Levin},
  title = {{GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.683},
}
Figure