The CoNLL-SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection

Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. McCarthy, Katharina Kann, S. J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, Mans HuldenCoNLL Shared Task 2018 · OpenReview

TL;DR

The CoNLL-SIGMORPHON 2018 Shared Task pushed the boundaries of morphological reinflection by introducing a new "cloze task" for contextual inflection and expanding the number of languages to 103. This work highlights the challenges and advancements in building robust morphological generation systems, particularly in low-resource settings, and provides valuable datasets for future research.

What Problem Does This Paper Address?

Morphological inflection, the process of changing a word's form to convey grammatical information (like tense, number, or case), is fundamental to natural language. While many languages have relatively simple inflectional systems, others, like Archi, can have thousands of forms for a single verb. Building computational systems that can accurately generate these inflected forms is crucial for tasks like machine translation, speech recognition, and natural language understanding. Previous shared tasks focused on generating inflected forms given a lemma and a set of features. This paper addresses the limitations of these tasks by:

  1. Expanding Language Coverage: The 2017 task covered 52 languages; the 2018 task significantly increased this to 103, demanding more generalizable models.
  2. Introducing Contextual Inflection: The traditional task ignored sentence context. The new "cloze task" requires systems to inflect a word form based on its surrounding words, mimicking human language acquisition and understanding. This is a much harder problem, requiring not just morphological knowledge but also semantic and pragmatic reasoning.
  3. Addressing Low-Resource Scenarios: Many languages have limited annotated data. The task included low, medium, and high data conditions to evaluate how well systems perform with varying amounts of training data, a critical challenge for real-world NLP applications.

Methods

The shared task featured two main tasks:

  • Task 1: Universal Morphological Reinflection: This task was similar to the previous year's, requiring systems to generate an inflected word form given a lemma and a "morphosyntactic description" (MSD) – a bundle of features. The key innovation here was the massive increase in language coverage (103 languages).
  • Task 2: Inflection in Context (Cloze Task): This novel task required systems to inflect a target lemma within a sentence. Participants were given the sentence with the target word masked and had to provide the contextually appropriate inflected form. Two tracks were offered:
  • Track 1: Provided context words, their lemmata, and MSDs.
  • Track 2: Provided only context words. In both tracks, the MSD for the target word was not provided, forcing systems to infer it from context.

Systems were evaluated on metrics like per-form accuracy and Levenshtein distance. Baselines included a simple lemma-copying system and a more sophisticated neural baseline using encoder-decoder architectures with attention, conditioned on various contextual information.

Datasets

  • Task 1: Utilized data from 103 typologically diverse languages. For most languages, data was sourced from Wiktionary, a large crowd-sourced dictionary. The data was structured as (lemma, feature bundle, inflected form) triples. Training, development, and test sets were sampled with varying data sizes (low, medium, high) to simulate low-resource conditions.
  • Task 2: Used Universal Dependencies (UD) v2 treebanks for seven languages (English, Finnish, French, German, Russian, Spanish, Swedish). The data was converted to the UniMorph format. Different supervision levels were provided across two tracks, with Track 1 offering more annotated context than Track 2.

Key Insights and Ideas

  • Context is King (for Cloze): Task 2, requiring contextual inflection, proved significantly harder than Task 1. Systems that could leverage sentence context to infer the correct MSD and inflected form performed much better, highlighting the importance of semantic and pragmatic understanding beyond pure morphological rules.
  • Low-Resource is Still Hard: While many systems improved over the previous year, especially in higher data conditions, performance in low-resource settings remained a significant challenge. This underscores the need for more robust generalization techniques.
  • Data Augmentation and Edit Sequences Help: Systems that incorporated strategies like learning edit sequences or generating artificial training data showed better performance, particularly in low-resource scenarios, suggesting these are effective ways to combat data sparsity.
  • Feature Combination vs. Ensemble: The analysis of oracle systems (which represent upper bounds on performance) showed that while ensembles of systems could achieve higher scores, the complementary strengths of different approaches were key. This suggests that combining diverse strategies is more effective than relying on a single, highly specialized method.
  • Original vs. Plausible Forms: For Task 2, evaluation on original UD forms and "plausible" forms (contextually appropriate but not necessarily the original) showed strong agreement in system rankings. This suggests that evaluating on original forms might be sufficient, simplifying future data annotation efforts.

Key Results

  • Task 1: 27 systems were submitted. The best systems showed improvements over the 2017 task, particularly in low-resource settings, with 41 out of 52 shared languages seeing gains.
  • Task 2: 6 systems were submitted. This task was significantly more challenging. The neural baseline system, conditioned on context, provided a strong benchmark. The UZH system was noted for its strong performance, especially in low-resource settings, often outperforming even the lemma-copying baseline.
  • Language Diversity: The task successfully covered 103 languages, demonstrating the potential for universal morphological models, though performance varied significantly across language families and typologies.
  • High vs. Low Resource: The gap between high and low-resource performance remained substantial, especially for Task 2.

Limitations and Future Work

  • Low-Resource Performance: While improvements were made, achieving high accuracy in low-resource settings, particularly for Task 2, remains a challenge.
  • Transfer Learning: The paper suggests exploring multilingual settings where models trained on high-resource languages can help low-resource ones, though disentangling true cross-language transfer from simple string copying is an open research question.
  • Data Creation: Creating more realistic datasets that reflect learner exposure (L1 or L2) is an important direction for future shared tasks.
  • IPA Representations: The authors suggest exploring orthographic representations beyond simple graphemes, such as IPA, to handle languages with idiosyncratic orthographies and potentially improve morphological analysis.
  • Semi-Supervised Learning: For Task 2, encouraging the use of external datasets for semi-supervised learning is proposed as a way to overcome the limitations of small training data.

Summary updated

Abstract

The CoNLL--SIGMORPHON 2018 shared task on supervised learning of morphological generation featured data sets from 103 typologically diverse languages. Apart from extending the number of languages involved in earlier supervised tasks of generating inflected forms, this year the shared task also featured a new second task which asked participants to inflect words in sentential context, similar to a cloze task. This second task featured seven languages. Task 1 received 27 submissions and task 2 received 6 submissions. Both tasks featured a low, medium, and high data condition. Nearly all submissions featured a neural component and built on highly-ranked systems from the earlier 2017 shared task. In the inflection task (task 1), 41 of the 52 languages present in last year's inflection task showed improvement by the best systems in the low-resource setting. The cloze task (task 2) proved to be difficult, and few submissions managed to consistently improve upon both a simple neural baseline system and a lemma-repeating baseline.
BibTeX
@inproceedings{cotterell2018conll,
  author = {Ryan Cotterell and Christo Kirov and John Sylak-Glassman and Géraldine Walther and Ekaterina Vylomova and Arya D. McCarthy and Katharina Kann and S. J. Mielke and Garrett Nicolai and Miikka Silfverberg and David Yarowsky and Jason Eisner and Mans Hulden},
  title = {{The CoNLL-SIGMORPHON 2018 Shared Task: Universal Morphological Reinflection}},
  booktitle = {CoNLL Shared Task},
  year = {2018},
  url = {https://openreview.net/forum?id=NGVvcBpn8PE},
}
Figure