GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text

Michael Ginn, Lindia Tjuatja, Taiqi He, Enora Rice, Graham Neubig, Alexis Palmer, Lori Levin

TL;DR

GlossLM introduces the largest standardized corpus of Interlinear Glossed Text (IGT) to date, covering over 450,000 examples across 1,800 languages. By continually pretraining a byte-level transformer (ByT5) on this massively multilingual data, the authors created a model that achieves state-of-the-art performance in automatic glossing, significantly lowering the barrier for documenting the world’s endangered languages.

What Problem Does This Paper Address?

Linguists documenting endangered languages often use Interlinear Glossed Text (IGT). As shown in Figure 1, IGT provides a morpheme-by-morpheme breakdown of a sentence, aligning the source language with grammatical labels (like "PAST" or "1PL") and a free translation.

Figure 1: Components of interlinear gloss with an Ara- paho sentence and English translation (Cowell, 2020). Blue boxes show transcriptions that are unsegmented (top) or segmented (bottom). Segmented text is split into morphemes which are aligned with the gloss labels shown in the green box. The task of automatic glossing uses some or all of the information in the gray box (tran- scription & translation) to generate the gloss line.

Despite its importance, IGT data is notoriously difficult to work with for machine learning. It is often trapped in PDFs or non-standardized formats, and glossing conventions vary wildly between researchers. Furthermore, most endangered languages have very little digitized data, making it nearly impossible to train traditional monolingual NLP models. The authors aim to solve this by creating a massive, standardized dataset and a pretrained model that leverages cross-lingual transfer to "help" low-resource languages.

Methods

The researchers took a two-pronged approach: data compilation and multilingual pretraining.

  1. Data Normalization: They aggregated IGT from six major sources and mapped the diverse glossing labels to the UniMorph schema, a standardized linguistic framework. This helps the model recognize that different labels (like SING, SG, or S) often refer to the same concept (Singular).
  2. GlossLM Model: They used ByT5, a byte-level version of the T5 transformer. Byte-level models are ideal here because they don't rely on a fixed vocabulary, allowing them to handle the diverse scripts and unique morphemes of 1,800 different languages.
  3. Continual Pretraining: Instead of training from scratch, they took a pretrained ByT5 model and "continually pretrained" it on their new IGT corpus. They framed the task as a text-to-text problem, using a prompt that includes the transcription, a flag for whether it's segmented, and the translation to predict the glosses.

Datasets

The authors compiled the GlossLM Corpus from six primary sources:

  • ODIN: Scraped from linguistic documents.
  • SIGMORPHON 2023 Shared Task: High-quality data for seven specific languages.
  • IMTVault: Extracted from LaTeX code in linguistics books.
  • APiCS: Data on pidgin and creole languages.
  • UraTyp: Uralic language data.
  • Guarani Corpus: A specific collection for the Guarani language.

The final dataset contains ~250k unique sentences, expanded to ~450k instances by creating unsegmented versions of segmented data.

Key Insights and Ideas

  • The Power of Cross-lingual Transfer: Even if the model hasn't seen a specific language before, pretraining on thousands of other languages helps it learn the "format" and "logic" of IGT. This is crucial for languages with only a few dozen examples.
  • Byte-Level is Best: Because many endangered languages use unique orthographies or lack standard tokenizers, working at the byte level avoids the "out-of-vocabulary" issues that plague word-level models.
  • Normalization Matters (Mostly): Mapping labels to UniMorph provided a significant boost for very low-resource languages (like Lezgi), though it had less impact on languages that already had substantial training data.
  • Zipfian Distribution: As seen in Figure 2, a few glosses (like PL for plural) are extremely common, while thousands of others appear only once. Standardizing the "head" of this distribution (the top 200 glosses) accounts for over 80% of all gloss instances in the corpus.
Figure 2: Distribution of unique glosses across all lan- guages.

Key Results

GlossLM achieved a new state-of-the-art (SOTA) in 5 out of 7 evaluation languages.

  • Zero-Shot/Few-Shot Success: The pretrained model (without language-specific fine-tuning) outperformed previous SOTA models on in-domain languages like Arapaho and Uspanteko (see Figure 3).
  • Low-Resource Gains: For languages with very little data, GlossLM significantly outperformed a standard ByT5 model. For example, on Lezgi, pretraining added a 15.1-point improvement in morpheme accuracy.
  • Scaling: As shown in Figure 5, the benefits of GlossLM's pretraining are most pronounced when the number of training examples is low (the left side of the graph). As data increases, the gap between GlossLM and a standard ByT5 narrows.
Figure 4: Morpheme accuracy for various systems.
Figure 5: Performance after monolingual finetuning, comparing a standard pretrained ByT5 with a continu- ally pretrained GLOSSLM model. The x-axis uses the log (base 10) of the number of training examples in a given language, for readability.

Limitations and Future Work

  • Data Imbalance: The corpus is heavily skewed toward a few languages (Arapaho makes up 20% of the data), which may bias the model.
  • Punctuation: The current evaluation ignores punctuation, which is important for practical documentation tools.
  • Domain Shift: The model struggles with highly technical or domain-specific language that differs from the linguistic papers it was trained on.
  • Future Directions: The authors suggest exploring other architectures beyond T5 and developing more robust ways to handle non-concatenative morphology (like infixes or tone changes).

Summary updated

Abstract

Language documentation projects often involve the creation of annotated text in a format such as interlinear glossed text (IGT), which captures fine-grained morphosyntactic analyses in a morpheme-by-morpheme format. However, there are few existing resources providing large amounts of standardized, easily accessible IGT data, limiting their applicability to linguistic research, and making it difficult to use such data in NLP modeling. We compile the largest existing corpus of IGT data from a variety of sources, covering over 450k examples across 1.8k languages, to enable research on crosslingual transfer and IGT generation. We normalize much of our data to follow a standard set of labels across languages. Furthermore, we explore the task of automatically generating IGT in order to aid documentation projects. As many languages lack sufficient monolingual data, we pretrain a large multilingual model on our corpus. We demonstrate the utility of this model by finetuning it on monolingual corpora, outperforming SOTA models by up to 6.6\%. Our pretrained model and dataset are available on Hugging Face.
BibTeX
@inproceedings{ginn2024glosslm,
  author = {Michael Ginn and Lindia Tjuatja and Taiqi He and Enora Rice and Graham Neubig and Alexis Palmer and Lori Levin},
  title = {{GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.683},
}
Figure