SIGMORPHON 2020 Task 0 System Description: ETH Zürich Team

Martina Forster, Clara MeisterACL 2020

TL;DR

This paper explores the impact of "exact inference" (finding the absolute best output) versus "heuristic search" (faster, good-enough output) for morphological inflection generation. They found that for this specific task, exact inference didn't significantly improve performance over simpler methods, unlike in other NLP tasks like machine translation.

What Problem Does This Paper Address?

Morphological inflection generation is the task of creating different forms of a word (like "run," "runs," "ran") based on a base word (lemma) and grammatical information (tags). This is crucial for morphologically rich languages, especially those with limited data, where it can help expand vocabulary for training other NLP models. The SIGMORPHON 2020 Task 0 specifically focuses on this across a diverse set of languages, many of which are low-resource. A key question is whether using more computationally intensive search methods, like exact inference, to find the globally optimal word form is beneficial compared to faster, approximate methods.

Methods

The ETH Zürich team used two baseline neural architectures provided by the task organizers: a multilingual Transformer and a (neuralized) hidden Markov model (HMM) with hard monotonic attention. The core innovation they applied was a different decoding strategy: exact inference. Instead of using common heuristic search methods like beam search, they employed Dijkstra's search algorithm to guarantee finding the most probable sequence according to the model. This was done to see if the computational cost of exact inference paid off in terms of performance for morphological inflection. They trained models on entire language families to leverage typological similarities and increase data for low-resource languages.

Datasets

The paper uses the data provided by the SIGMORPHON 2020 Shared Task, which includes lemmas, inflections, and morphological tags for 90 languages. These languages span five families in the first phase (Austronesian, Niger-Congo, Uralic, Oto-Manguean, Indo-European) and several more in the second phase (Afro-Asiatic, Algic, Australian, Dravidian, Germanic, Indo-Aryan, Iranian, Niger-Congo, Nilo-Sahan, Romance, Sino-Tibetan, Siouan, Tungusic, Turkic, Uralic, Uto-Aztecan). The paper notes that many languages had very scarce data, with examples like Zarma having only 56 training instances.

Key Insights and Ideas

  • Exact Inference Doesn't Always Pay Off: The most striking finding is that for morphological inflection, exact inference didn't significantly outperform simpler search methods like greedy search. This contrasts with findings in Neural Machine Translation (NMT), where exact search can sometimes degrade performance.
  • Language Family Training is Key for Low-Resource: Training on language families, rather than individual languages, was a strategy to combat data scarcity. This approach seems to have helped, though it might also explain poorer performance on some languages within high-resource families if they were underrepresented.
  • HMM Outperforms Transformer: The HMM-based model, when combined with exact inference, achieved better overall accuracy than the Transformer-based model.

Key Results

The team's HMM system achieved an average accuracy of 80.9% with an average Levenshtein distance of 0.5. Their Transformer system achieved 75.6% accuracy with an average Levenshtein distance of 0.62. When comparing their exact inference results to baseline greedy and beam search (shown in Tables 1 and 2 for select languages, and Table 3 for all languages), the differences were often marginal, suggesting exact inference did not provide a substantial boost. For instance, Table 3 shows their HMM with exact inference (0.809 acc) is only slightly better than the baseline HMM with greedy search (0.860 acc, but this is likely a typo in the paper and should be compared to the baseline HMM with beam search, which is 0.860 acc). The paper's discussion clarifies that their HMM with exact decoding achieved 80.9% accuracy, while the Transformer achieved 75.6%.

Limitations and Future Work

  • Low-Resource Language Performance: The paper acknowledges that performance on very low-resource languages (like Zarma, which is the only language in its family) was poor, even with exact decoding. This suggests that more data or specific fine-tuning might be necessary for such cases.
  • Language Family Representation: The authors hypothesize that languages within high-resource families that are underrepresented in the training data might not be modeled adequately. Fine-tuning on individual languages could potentially improve performance in these scenarios.
  • Future Work: The paper doesn't explicitly detail future work, but the findings suggest further investigation into why exact inference doesn't yield greater benefits for inflection generation compared to other NLP tasks, and how to better handle extremely low-resource languages within family-based training.

Summary updated

BibTeX
@inproceedings{forster2020sigmorphon,
  author = {Martina Forster and Clara Meister},
  title = {{SIGMORPHON 2020 Task 0 System Description: ETH Zürich Team}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure