LISN @ SIGMORPHON 2023 Shared Task on Interlinear Glossing

Shu Okabe, François YvonACL 2023

TL;DR

This paper presents LISN's submission to the SIGMORPHON 2023 shared task on interlinear glossing. They adapt a statistical machine translation system (Lost) to the task of automatically generating interlinear glosses, effectively handling the challenge of unknown lexical glosses by leveraging target language translations. Their approach achieves competitive results, particularly in low-resource settings.

What Problem Does This Paper Address?

Interlinear glossing is a crucial technique in linguistics for annotating text with morpheme-level information, breaking down words into their constituent parts and assigning a meaning or grammatical function to each. The SIGMORPHON 2023 shared task focuses on automating this process. A major challenge in automatic glossing is the "open vocabulary" problem for lexical morphemes: the set of possible glosses for word meanings is not fixed and can be vast, especially when dealing with languages with rich morphology or limited annotated data. Traditional sequence labeling models struggle when the label set is not predefined. This paper tackles this challenge by developing a system that can infer and assign novel lexical glosses by utilizing information from a parallel translation.

Methods

The core of LISN's system is an adaptation of "Lost," a probabilistic model originally designed for statistical machine translation, which is based on Conditional Random Fields (CRFs). The authors frame automatic glossing as a sequence labeling task at the morpheme level.

Key methodological contributions include:

  • Leveraging Target Translations for Lexical Glosses: The system assumes that lexical glosses can be inferred from a parallel translation of the source sentence. The set of potential labels for a morpheme is constructed by combining:
  • Known grammatical glosses.
  • Lemmas from the target translation.
  • Frequently-associated labels from the training data.
  • Restricted Search Space: To manage the potentially huge label space, Lost's ability to define a local or sentence-specific set of labels is crucial. This significantly reduces computational complexity during training and inference.
  • Automatic Alignment: To link lexical glosses to words in the target translation, they employ SimAlign, a neural word alignment model. This alignment is performed at the morpheme level.
  • Feature Engineering: The system utilizes a rich set of features, including:
  • Basic unigram and bigram features of the gloss itself.
  • Features incorporating the predicted grammatical/lexical category (b) and the projected Part-of-Speech (PoS) tag (p) of the aligned target word.
  • Additional features in their S2 submission, such as a "copy" feature for identical words and a positional feature to capture source-target alignment information.

Datasets

The authors used the provided training and development datasets for the SIGMORPHON 2023 shared task. The languages included in their experiments were Tsez (ddo), Gitksan (git), Lezgi (lez), Natugu (ntu), and Uspanteko (usp). They note that they could not run their models on Nyangbo (nyb) due to the reliance on translations for lexical glosses, which were not available for that language in the provided dataset.

Key Insights and Ideas

  • Translation as a Lexical Oracle: The core insight is that a parallel translation, even if not perfectly aligned, provides a rich source of information for inferring lexical meanings of morphemes. This bypasses the need for a pre-compiled, exhaustive lexicon.
  • CRF Flexibility for Open Vocabulary: The adaptation of a CRF-based system (Lost) with a dynamically defined, sentence-specific label set is key to handling the open-ended nature of lexical glosses.
  • Grammatical vs. Lexical Glosses: The paper highlights that grammatical glosses (like tense, case) are generally easier to predict than lexical ones, as their label set is finite and well-defined. Lexical glosses benefit more from increased training data.
  • Feature Engineering Matters: The inclusion of features related to the predicted category (grammatical/lexical) and PoS tags of aligned target words significantly improves performance, suggesting that richer linguistic context is beneficial.

Key Results

LINS's systems (S1 and S2) consistently outperformed the provided baseline across multiple languages and metrics (word accuracy, morpheme accuracy, BLEU, etc.).

  • Significant Improvement: For example, on the development dataset, S1 achieved morpheme accuracy of 91.0% for Tsez, compared to the baseline's 85.0%. For Gitksan, S1 achieved 55.5% morpheme accuracy, a substantial jump from the baseline's 30.0%.
  • Low-Resource Performance: The improvements were particularly pronounced in low-resource languages like Gitksan and Lezgi, demonstrating the system's robustness.
  • S2 vs. S1: The S2 system, with its additional features, offered a slight but consistent improvement over S1, suggesting the value of features capturing word copying and positional alignment.
  • Training Data Impact: Lexical gloss prediction significantly benefits from larger training datasets, while grammatical glosses remain relatively stable.

Limitations and Future Work

  • Dependency on Translations: The system's reliance on parallel translations for lexical glosses means it cannot be applied to languages without such resources (like Nyangbo in this task).
  • Alignment Noise: The automatic alignment process, while effective, can introduce noise. The authors suggest future work to model alignment as an unobserved variable within a latent variable model to potentially remove this dependency.
  • Neural Embeddings: While their current approach uses BERT embeddings, they acknowledge that for very low-resource settings, neural methods might be data-intensive. Future work could explore integrating word embeddings for better-resourced languages.
  • Exploration of Features: The authors plan to further explore feature functions on both source and target sides.

Summary updated

BibTeX
@inproceedings{okabe2023lisn,
  author = {Shu Okabe and François Yvon},
  title = {{LISN @ SIGMORPHON 2023 Shared Task on Interlinear Glossing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure