Lightweight morpheme labeling in context: Using structured linguistic representations to support linguistic analysis for the language documentation context

Bhargav Shandilya, Alexis PalmerACL 2023

TL;DR

This paper introduces a "gloss-to-gloss" (g2g) model for lightweight morphological analysis, designed to be computationally efficient and data-light for endangered language documentation. By framing morpheme labeling as a translation task and leveraging linguistic dimensions, their CNN-based model achieves performance comparable to larger transformer models at a fraction of the computational cost.

What Problem Does This Paper Address?

Documenting and analyzing endangered and less-studied languages is crucial for linguistic preservation, but it's a labor-intensive process. A key output is Interlinear Glossed Text (IGT), which requires detailed morphological analysis (identifying morphemes and their functions). Modern NLP tools for this task, like those based on large transformer models, demand vast amounts of annotated data and significant computational resources, often beyond the reach of researchers and language communities working with low-resource languages. This paper aims to bridge this gap by providing a lightweight, efficient tool that can assist in the IGT creation process, saving valuable expert time.

Methods

The core of the paper is the "gloss-to-gloss" (g2g) model. It treats morphological analysis as a translation problem: transforming source language morpheme labels into target language morpheme labels.

  • Pseudo-parallel Data Generation: Since true parallel data with aligned morpheme labels is scarce, the authors create "pseudo-parallel" data. They back-translate high-resource language texts (e.g., English) into the target language and then automatically label the English text with a UniMorph-trained morphological labeler.
  • Structured Linguistic Representation: To incorporate linguistic knowledge and reduce the label space, morpheme labels are mapped to broader "linguistic dimensions" (LDs), as defined by the UniMorph schema. This reduces the number of distinct labels by about 60%.
  • CNN Architecture: The model uses a convolutional neural network (CNN) to process the data. The input is a 2D "heat map" representation of the morpheme labels (and word order), treating the task as an image-to-image translation problem. This CNN generates heat maps predicting output gloss labels.
  • Lexical Information Integration: Word embeddings (w2v) for the source language text are concatenated with the CNN's output to capture lexical semantics, feeding into a shallow three-layer network for final label prediction.
  • Probabilistic Length Modeling: To handle variable sentence lengths, a probabilistic model is used to drop low-probability morpheme combinations, ensuring the output matches the actual sentence length.
  • Baseline Comparison: The CNN model is compared against a fine-tuned BERT model, a state-of-the-art transformer, to demonstrate its efficiency and comparable performance.

Datasets

  • SIGMORPHON 2019 Shared Task Data: Used for initial model development and testing. This dataset comprises sentence-level morphological analysis in the UniMorph style for 68 languages. The authors report results on 9 diverse languages, using a 60/20/20 train/dev/test split.
  • Uspanteko Data: A true low-resource language dataset from a language documentation project (Pixabaj et al., 2007) is used for final evaluation. This data required custom mapping from its original IGT format to UniMorph.
  • English UniMorph Data: Used to train the morphological labeler for generating pseudo-parallel English data.

Key Insights and Ideas

  • Framing as Translation: Treating morpheme labeling as a "gloss-to-gloss" translation task is effective, especially when direct parallel data is unavailable.
  • Linguistic Dimensions as a Bridge: Mapping morpheme labels to linguistic dimensions significantly simplifies the problem and incorporates valuable linguistic knowledge, improving efficiency and accuracy.
  • CNNs for Linguistic Structure: CNNs, often associated with image processing, can be effectively applied to structured linguistic data like morphological labels, treating them as a form of "image."
  • Efficiency is Key for Documentation: The paper strongly advocates for computationally lightweight models, recognizing the practical constraints faced by researchers in low-resource settings. The dramatic reduction in training time is a major selling point.

Key Results

  • Comparable Performance: The lightweight CNN model achieves performance comparable to a fine-tuned BERT model across various high-resource languages, often within 5 percentage points in F1 scores.
  • Significant Speedup: The CNN model trains in minutes on a standard laptop (average 8.5 minutes), while the BERT baseline requires hours on a multi-GPU cluster (average 3.5 hours).
  • Uspanteko Performance: The model performs comparably on the low-resource Uspanteko language as it does on morphologically complex high-resource languages like Turkish, suggesting its utility for actual language documentation.
  • Error Analysis: For Uspanteko, the model makes more errors in categories like case, person, and number, potentially due to differences with English (e.g., ergative-absolutive case system). "Unknown" labels are a significant source of false positives, indicating areas where human experts are most needed.

Limitations and Future Work

  • Pseudo-Parallel Data Quality: The use of machine translation for creating pseudo-parallel data introduces compounding errors, which can affect the final gloss output.
  • Unordered Labels: The current model outputs an unordered set of labels for a word, unlike typical IGT which maintains order.
  • Variability in IGT Schemes: The model currently maps to UniMorph. Future work aims to mimic the specific glossing conventions of individual documentation projects, including label order and syntax.
  • Ethical Considerations: The authors acknowledge the risk of perpetuating existing oppression when working with NLP for endangered languages. They emphasize a cautious approach, starting with non-threatened languages and involving community members only when the system is robust.
  • Further Linguistic Knowledge Integration: Exploring more sophisticated ways to incorporate linguistic knowledge is a planned future direction.
  • Field Linguist Evaluation: Evaluating the model's usability and deployment with field linguists and language community members is a crucial next step.

Summary updated

BibTeX
@inproceedings{shandilya2023lightweight,
  author = {Bhargav Shandilya and Alexis Palmer},
  title = {{Lightweight morpheme labeling in context: Using structured linguistic representations to support linguistic analysis for the language documentation context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure