TL;DR
GlossLM introduces the largest standardized corpus of Interlinear Glossed Text (IGT) to date, covering over 450,000 examples across 1,800 languages. By continually pretraining a byte-level transformer (ByT5) on this massively multilingual data, the authors created a model that achieves state-of-the-art performance in automatic glossing, significantly lowering the barrier for documenting the world’s endangered languages.
What Problem Does This Paper Address?
Linguists documenting endangered languages often use Interlinear Glossed Text (IGT). As shown in Figure 1, IGT provides a morpheme-by-morpheme breakdown of a sentence, aligning the source language with grammatical labels (like "PAST" or "1PL") and a free translation.
Despite its importance, IGT data is notoriously difficult to work with for machine learning. It is often trapped in PDFs or non-standardized formats, and glossing conventions vary wildly between researchers. Furthermore, most endangered languages have very little digitized data, making it nearly impossible to train traditional monolingual NLP models. The authors aim to solve this by creating a massive, standardized dataset and a pretrained model that leverages cross-lingual transfer to "help" low-resource languages.
Methods
The researchers took a two-pronged approach: data compilation and multilingual pretraining.
- Data Normalization: They aggregated IGT from six major sources and mapped the diverse glossing labels to the UniMorph schema, a standardized linguistic framework. This helps the model recognize that different labels (like
SING,SG, orS) often refer to the same concept (Singular). - GlossLM Model: They used ByT5, a byte-level version of the T5 transformer. Byte-level models are ideal here because they don't rely on a fixed vocabulary, allowing them to handle the diverse scripts and unique morphemes of 1,800 different languages.
- Continual Pretraining: Instead of training from scratch, they took a pretrained ByT5 model and "continually pretrained" it on their new IGT corpus. They framed the task as a text-to-text problem, using a prompt that includes the transcription, a flag for whether it's segmented, and the translation to predict the glosses.
Datasets
The authors compiled the GlossLM Corpus from six primary sources:
- ODIN: Scraped from linguistic documents.
- SIGMORPHON 2023 Shared Task: High-quality data for seven specific languages.
- IMTVault: Extracted from LaTeX code in linguistics books.
- APiCS: Data on pidgin and creole languages.
- UraTyp: Uralic language data.
- Guarani Corpus: A specific collection for the Guarani language.
The final dataset contains ~250k unique sentences, expanded to ~450k instances by creating unsegmented versions of segmented data.
Key Insights and Ideas
- The Power of Cross-lingual Transfer: Even if the model hasn't seen a specific language before, pretraining on thousands of other languages helps it learn the "format" and "logic" of IGT. This is crucial for languages with only a few dozen examples.
- Byte-Level is Best: Because many endangered languages use unique orthographies or lack standard tokenizers, working at the byte level avoids the "out-of-vocabulary" issues that plague word-level models.
- Normalization Matters (Mostly): Mapping labels to UniMorph provided a significant boost for very low-resource languages (like Lezgi), though it had less impact on languages that already had substantial training data.
- Zipfian Distribution: As seen in Figure 2, a few glosses (like PL for plural) are extremely common, while thousands of others appear only once. Standardizing the "head" of this distribution (the top 200 glosses) accounts for over 80% of all gloss instances in the corpus.
Key Results
GlossLM achieved a new state-of-the-art (SOTA) in 5 out of 7 evaluation languages.
- Zero-Shot/Few-Shot Success: The pretrained model (without language-specific fine-tuning) outperformed previous SOTA models on in-domain languages like Arapaho and Uspanteko (see Figure 3).
- Low-Resource Gains: For languages with very little data, GlossLM significantly outperformed a standard ByT5 model. For example, on Lezgi, pretraining added a 15.1-point improvement in morpheme accuracy.
- Scaling: As shown in Figure 5, the benefits of GlossLM's pretraining are most pronounced when the number of training examples is low (the left side of the graph). As data increases, the gap between GlossLM and a standard ByT5 narrows.
Limitations and Future Work
- Data Imbalance: The corpus is heavily skewed toward a few languages (Arapaho makes up 20% of the data), which may bias the model.
- Punctuation: The current evaluation ignores punctuation, which is important for practical documentation tools.
- Domain Shift: The model struggles with highly technical or domain-specific language that differs from the linguistic papers it was trained on.
- Future Directions: The authors suggest exploring other architectures beyond T5 and developing more robust ways to handle non-concatenative morphology (like infixes or tone changes).
Summary updated