Leveraging Principal Parts for Morphological Inflection

Ling Liu, Mans HuldenACL 2020

TL;DR

This paper tackles the challenge of morphological inflection, specifically generating correct word forms from lemmas and descriptions, especially in low-resource languages. They propose a novel approach that leverages the linguistic concept of "principal parts" of a paradigm to improve Transformer-based models, achieving state-of-the-art results on the SIGMORPHON 2020 shared task.

What Problem Does This Paper Address?

Morphological inflection is the process of changing a word's form to indicate grammatical information like tense, number, or case (e.g., "walk" -> "walks," "walking," "walked"). This is a fundamental task in natural language processing, particularly for morphologically rich languages. While sequence-to-sequence models like Transformers have shown great success, their performance can degrade significantly when training data is scarce (low-resource languages). The SIGMORPHON 2020 shared task specifically aimed to test systems on their ability to generalize to new languages, including those with limited training data. The paper addresses the gap of effectively handling morphological inflection in low-resource settings by incorporating linguistic knowledge.

Methods

The core of the system is the Transformer architecture, a standard for sequence-to-sequence tasks. The novelty lies in how they adapt the input and leverage linguistic insights:

  • Paradigm Cell Filling: They reframe the inflection task as filling missing "cells" in a linguistic paradigm. A paradigm represents all the inflected forms of a single word (lexeme).
  • Principal Parts: They utilize the linguistic concept of "principal parts," which are a minimal set of inflected forms from which all other forms in a paradigm can be derived. This is crucial because the lemma (the base form) isn't always the most informative starting point.
  • 1-src and 2-src Models:
    • The 1-src model takes one known slot (lexeme + MSD) from a paradigm as input and predicts another known slot. This is done by creating multiple input-output pairs from each paradigm, where different known slots are used as sources to predict the target slot.
  • The 2-src model extends this by using combinations of two known slots as input to predict the target slot. This is particularly useful for languages with very limited data.
  • Prediction Selection: When multiple predictions are generated for a single target form (due to different input combinations), they select the best one using either majority vote or by averaging log-likelihood scores.

Datasets

The primary dataset used is from the SIGMORPHON 2020 shared task 0. This dataset comprises 90 languages from 18 language families, with varying amounts of training data. Languages are categorized as high-resource (>= 5,000 training triples) and low-resource (< 5,000 training triples). The paper focuses its novel methods on the low-resource development languages.

Key Insights and Ideas

  • Principal Parts are Key for Low-Resource Settings: The paper strongly suggests that the lemma alone is insufficient for robust morphological inflection, especially when data is limited. Leveraging the concept of principal parts, which are linguistically motivated, significantly improves performance.
  • Transformer Generalizes Well, but Needs Guidance: The Transformer architecture itself is powerful and can generalize to new languages. However, providing it with structured linguistic information (like paradigm relationships) enhances its ability to learn effectively, even with sparse data.
  • Paradigm Completeness Matters: The more complete a reconstructed paradigm is (i.e., the more known slots it contains), the better the models perform. This reinforces the idea that having more related forms available helps the model learn the inflectional patterns.
  • Uralic Languages are Challenging: The paper notes that Uralic languages are particularly difficult for their models, likely due to their complex inflectional systems and potentially more incomplete paradigms.

Key Results

The authors' submission achieved the first rank in both average accuracy and Levenshtein distance among all submissions to the SIGMORPHON 2020 shared task 0.

  • The 1-src model achieved an average accuracy of 91.72% on the development data for 24 low-resource languages, which was 1.3% higher than the unaugmented Transformer baseline and 0.55% higher than the best overall baseline.
  • The 2-src model showed even better performance on 7 languages with fewer than 2,000 training examples, achieving an average accuracy of 91.63%.
  • Their submission had the lowest standard deviation for accuracy and the second lowest for edit distance across all 90 languages.

Limitations and Future Work

  • Data Scarcity Remains a Challenge: While their methods improve performance on low-resource languages, the paper acknowledges that performance still tends to decrease as training data becomes scarcer.
  • Automatic Principal Part Identification: The current approach relies on reconstructing paradigms and implicitly using principal parts. Future work could explore explicitly identifying principal parts for languages or tuning the number and choice of source forms on a per-language basis.
  • Exploration of More Complex Input Combinations: The paper mentions experimenting with 2-, 3-, and 4-random-src models, but found that the 1-src and 2-src models offered the best balance of performance and complexity. Further research could investigate more sophisticated ways to combine information from multiple slots.

Summary updated

BibTeX
@inproceedings{liu2020leveraging,
  author = {Ling Liu and Mans Hulden},
  title = {{Leveraging Principal Parts for Morphological Inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure