The SIGMORPHON 2020 Shared Task on Unsupervised Morphological Paradigm Completion

Katharina Kann, Arya D. McCarthy, Garrett Nicolai, Mans HuldenACL 2020

TL;DR

This paper introduces the SIGMORPHON 2020 Shared Task 2, a novel challenge in unsupervised morphological paradigm completion. The goal was to generate all inflected forms of a lemma given only a corpus and a list of lemmas, without any explicit morphological annotations. Surprisingly, no submitted system outperformed the provided baseline, highlighting that this is still a significant open challenge in NLP.

What Problem Does This Paper Address?

Morphologically rich languages encode grammatical information directly into words through inflection. For example, in English, "walk" can become "walks," "walked," or "walking." Humans learn to do this naturally, even for novel words. However, building computational models that can generate these complete sets of inflected forms (paradigms) for any given lemma, without explicit training data on these forms, is a difficult problem. This is particularly relevant for low-resource languages where annotated data is scarce. The SIGMORPHON 2020 Task 2 aimed to push the boundaries of unsupervised morphological paradigm completion, mimicking a realistic scenario where only a raw text corpus and a dictionary of base forms (lemmas) are available.

Methods

The core of the task involved unsupervised learning, meaning systems had to infer morphological rules and paradigms from raw text. The SIGMORPHON 2020 Task 2 had a baseline system that acted as a benchmark. This baseline system was a pipeline consisting of two main components:

  • Retrieval Component: This part of the system scanned the provided corpus (the JHU Bible Corpus) to identify potential inflected forms and their corresponding lemmas. It used techniques like "edit trees" to find common patterns of word formation (e.g., adding "-ed" to form the past tense). It also aimed to retrieve additional lemmas to gather more evidence for these patterns.
  • Generation Component: Once potential inflected forms and their associated lemma-form-slot triplets were identified, this component used them to train an inflectional generator. This generator then predicted new, unobserved forms for the complete morphological paradigm of each input lemma.

Submitted systems generally fell into two categories:

  • Retrieval+X: These systems leveraged the baseline's retrieval component but substituted their own custom inflectional generation models.
  • Segment+Conquer: These systems took a different approach, often involving character segmentation to identify stem-suffix splits and then using language models to score potential inflections.

Datasets

The task utilized the JHU Bible Corpus, which contains text in over 1600 languages. For the shared task, data was provided for 5 development languages (Maltese, Persian, Portuguese, Russian, Swedish) and 9 test languages (Basque, Bulgarian, English, Finnish, German, Kannada, Navajo, Spanish, Turkish). The development languages were available for system tuning, while the test languages were revealed shortly before the deadline, simulating a real-world scenario with limited prior knowledge. Morphological information for evaluation was sourced from UniMorph, a resource containing paradigms for over 100 languages, but participants only had access to this for the development languages.

Key Insights and Ideas

  • Unsupervised Paradigm Completion is Hard: The most striking finding is that no submitted system managed to outperform the baseline system on average across all test languages. This strongly suggests that unsupervised morphological paradigm completion remains a significant open challenge.
  • Baseline Effectiveness: The baseline system, despite its modular pipeline approach, proved remarkably effective, setting a high bar for participants. This highlights the power of combining retrieval from a corpus with a generative inflection model.
  • Retrieval+X Outperforms Segment+Conquer: Systems that built upon the baseline's retrieval mechanism generally performed better than those using a "Segment+Conquer" strategy. This indicates that effectively leveraging existing text data for identifying potential inflections is crucial.
  • Unseen Lemmas are a Challenge: While the performance difference between seen and unseen lemmas wasn't drastically large, some systems struggled significantly with unseen lemmas, producing nonsensical outputs. This suggests that the ability to generalize to novel words is a key differentiator.
  • Contextual Variety Matters: The paper suggests that systems need to compensate for low contextual variety in their training data, rather than the inherent difficulty of unseen lemmas being the primary issue.

Key Results

  • No System Beat the Baseline: On average, across all 9 test languages, no submitted system achieved a higher Macro-averaged Best-Match Accuracy (BMAcc) than the baseline.
  • Individual Language Successes: While no system won overall, two "Retrieval+X" systems managed to achieve the best results on 3 individual languages.
  • BMAcc as Evaluation Metric: The Macro-averaged Best-Match Accuracy (BMAcc) was used to evaluate systems. It measures the accuracy of correctly identifying and grouping inflected forms into paradigm slots, even when the slot labels themselves are arbitrary.
  • Development vs. Test Performance: Results on development languages provided insights but did not count towards the final ranking. The official evaluation was on the surprise test languages.

Limitations and Future Work

  • Unsupervised Nature: The task's inherent difficulty lies in its unsupervised nature, requiring systems to infer complex morphological rules without explicit supervision.
  • Limited Data for Some Languages: For some languages, UniMorph coverage was limited, impacting the number of lemmas available for the task.
  • Future Directions: The authors suggest extending the task to more languages and parts of speech, exploring derivational morphology, and potentially integrating it with traditional inflectional tasks. A particularly exciting future direction is incorporating speech data, which aligns more closely with how human L1 learners acquire language. The release of the data, baseline, and evaluation script aims to foster further research in this area.

Summary updated

BibTeX
@inproceedings{kann2020sigmorphon,
  author = {Katharina Kann and Arya D. McCarthy and Garrett Nicolai and Mans Hulden},
  title = {{The SIGMORPHON 2020 Shared Task on Unsupervised Morphological Paradigm Completion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure