Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages

Micha Elsner, David LiuNAACL 2025

TL;DR

This paper introduces a novel word-by-word prompting strategy for Large Language Models (LLMs) to automate interlinear glossed text (IGT) creation for low-resource languages. Their approach, which leverages retrieval and linguistic instructions, significantly outperforms baseline methods and shows promise for making linguistic documentation more accessible.

What Problem Does This Paper Address?

Documenting and preserving the world's vast linguistic diversity is a race against time, with many languages facing endangerment. Interlinear glossed text (IGT) is a crucial tool for linguistic documentation, breaking down sentences into word-by-word grammatical analyses and translations. However, creating IGT is time-consuming and resource-intensive, hindering efforts to document endangered languages. While automated methods exist, they often rely on specialized, task-specific models. This paper explores how general-purpose LLMs, with their ability to follow natural language instructions, can be leveraged to create IGT more efficiently and accessibly, particularly for languages with limited existing data.

Methods

The core of this work is a "word-by-word retrieval-based prompting" approach for LLMs. Instead of prompting the LLM with entire sentences, they prompt it for each word individually. This involves:

  • Retrieval: For each target word, the system retrieves similar sentences from a training corpus. This includes exact matches and approximate matches based on shared substrings.
  • Contextualization: The retrieved sentences, along with the sentence translation and the target word's context, are fed into the LLM prompt.
  • Instruction Following: Crucially, the LLM is instructed to follow Leipzig glossing conventions and output a JSON format for the top three glosses.
  • Instruction Generation: For particularly challenging cases (like syncretic forms), they use the LLM itself to generate linguistic instructions that help disambiguate tag pairs. These generated instructions are then incorporated into the prompting process.

This contrasts with previous LLM approaches that might use whole-sentence prompting or rely solely on retrieval without explicit linguistic instructions.

Datasets

The research utilizes the datasets from the SIGMORPHON 2023 shared task on interlinear glossing. This task provided data for seven languages: Arapaho, Gitksan, Lezgi, Natugu, Nyangbo, Tsez, and Uspanteko. The authors specifically focused their development and prompt tuning on Tsez due to its moderate size. They used the Track 1 data files, which do not include pre-existing morphological segmentation.

Key Insights and Ideas

  • Word-by-word prompting is effective: Breaking down the glossing task into word-level prompts allows LLMs to perform comparably to sentence-level approaches, and in some cases, more efficiently.
  • LLMs can follow linguistic instructions: The ability of LLMs to interpret and apply natural language instructions is a significant advantage for tasks requiring nuanced grammatical analysis, reducing errors on complex features.
  • A "3-best oracle" is powerful: Even when the LLM's top gloss prediction isn't perfect, offering the top three options to a human annotator significantly increases the chances of finding a correct gloss, outperforming even the challenge winner in many cases.
  • Instruction generation can solve specific problems: LLMs can be used to create targeted linguistic rules to disambiguate confusing grammatical phenomena, like syncretic morphemes, leading to improved accuracy.

Key Results

  • The word-by-word retrieval-based LLM approach beats the BERT-based shared task baseline for every language in the morpheme-level score category.
  • A 3-best oracle system achieved higher word-level scores than the challenge winner (a tuned sequence model) in five out of seven languages.
  • In a case study on Tsez, automatically generated linguistic instructions reduced errors by 10% for a specific confusing grammatical feature (syncretic verb forms).
  • The system's performance was comfortably ahead of the SIGMORPHON baseline in word-level scores for most languages, though slightly behind the challenge winner.

Limitations and Future Work

  • Cost: Running LLMs, especially for extensive experiments, is computationally expensive. This limited the scope of their experiments, preventing rigorous ablation studies and comparisons with other LLMs.
  • Reproducibility: The use of a proprietary LLM (GPT-4) limits reproducibility. The authors suggest that future work could explore open-source models.
  • Data Leakage: While they tested for memorization and found no evidence of verbatim recall of test sentences, the authors acknowledge the inherent challenge of data leakage in LLM research.
  • Expertise: The authors note their lack of expertise in Caucasian languages, which might influence their interpretation of GPT-4's generated instructions.
  • Future Work: The authors suggest exploring multistep "chains of thought" for more complex annotation tasks, applying instruction generation to other morphological disambiguation problems, and investigating what distinguishes successful LLM instruction following from failures in translation and glossing. They also emphasize the potential for LLM-based systems to be more accessible to users than traditional sequence learners.

Summary updated

BibTeX
@inproceedings{elsner2025prompt,
  author = {Micha Elsner and David Liu},
  title = {{Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure