An Ensembled Encoder-Decoder System for Interlinear Glossed Text

Edith CoatesACL 2023

TL;DR

This paper presents an ensembled encoder-decoder system for automatically generating interlinear glosses (IGT) for under-resourced languages. The system achieves competitive results by combining multiple models trained with different window sizes and leveraging an LSTM architecture for efficiency, demonstrating a practical approach to a challenging linguistic task.

What Problem Does This Paper Address?

Interlinear glossed text (IGT) is a crucial linguistic annotation that breaks down sentences into their constituent morphemes, providing their grammatical function and translation. This is invaluable for linguistic documentation and analysis, especially for under-resourced languages where manual annotation is time-consuming and expensive. However, automatically generating accurate IGT is challenging due to the variability in annotation styles and the complexity of morphological structure. This paper tackles the problem of building an effective IGT generation system for the SIGMORPHON 2023 shared task, focusing on the "closed track" where only the source sentence, its gloss, and translation are available for training.

Methods

The core of the system is an encoder-decoder architecture based on LSTMs (Long Short-Term Memory networks). The authors chose LSTMs over Transformers due to computational constraints, as training Transformers on their available hardware was too resource-intensive.

Key methodological aspects include:

  • Ensembling: The system combines predictions from multiple models. Specifically, they trained two models: one using a word window of size one (considering only the target word) and another using a window of size two (considering the target word and its immediate neighbors). During inference, the system selects the prediction with the lowest negative log-likelihood (NLL) score from these models.
  • Word-level Training Examples: Instead of processing entire sentences, the system was trained on individual words, providing context from adjacent words. This allows for sentence-level results to be constructed by concatenating word-level predictions.
  • Output Representation: The authors experimented with different ways to represent the target gloss. They ultimately settled on a token-level output alphabet, which proved more effective than a character-level one. They also explored using special tokens for stems but found that using the original translated stems from the dataset yielded better results.
  • Stem Generation (Exploratory): They also experimented with a separate sequence-to-sequence model to generate stems from the translation, but this approach did not significantly improve overall performance.

Datasets

The system was trained and evaluated on data from the SIGMORPHON 2023 shared task on interlinear glossed text. This dataset includes examples from various languages, with the paper specifically mentioning experiments on Tsez and Gitksan (for which they also developed an inflectional database). The task's "closed track" provided only the source sentence, gloss, and translation.

Key Insights and Ideas

  • LSTM's Practicality: For resource-constrained environments, LSTMs can still provide a strong baseline for complex sequence-to-sequence tasks like IGT generation, offering a more accessible alternative to larger Transformer models.
  • Ensembling Benefits: Combining models trained with different contextual windows (word window sizes) demonstrably improves performance, suggesting that capturing varied local context is beneficial.
  • Token-Level Output Advantage: Representing the gloss as a sequence of tokens, rather than characters, appears to be a more effective strategy for this task.
  • Morpheme-to-Word Ratio Matters: The paper hypothesizes that the performance differences across languages might be linked to their morpheme-to-word ratios, with higher ratios potentially benefiting from the system's approach.

Key Results

The ensembled system consistently outperformed the shared task's baseline model in terms of both word-level and morpheme-level accuracy across most languages. For example, on the Nyangbo language, their system achieved a 72% higher word-level accuracy than the baseline. However, performance varied significantly by language; for Arapaho, their system performed 15% worse than the baseline. The paper also notes that their system was relatively inexpensive to train, runnable on a single laptop.

Limitations and Future Work

The authors acknowledge that time constraints prevented a thorough grid search of hyperparameters, preprocessing techniques, and stem approaches, suggesting that a more optimal system might exist.

Future work could explore:

  • Ensembling and Data Augmentation: Investigating these techniques further for CRF or LSTM-based token classification systems.
  • Stem Generation: Developing more sophisticated stem generation systems, potentially using linguist-created inflectional databases or upstream word alignment models.
  • Low-Resource Systems: Continuing research into low-resource systems for automatic IGT, considering both computational efficiency and dataset size.

Summary updated

BibTeX
@inproceedings{coates2023ensembled,
  author = {Edith Coates},
  title = {{An Ensembled Encoder-Decoder System for Interlinear Glossed Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure