CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages

Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, Mans Hulden

Abstract

The CoNLL-SIGMORPHON 2017 shared task on supervised morphological generation required systems to be trained and tested in each of 52 typologically diverse languages. In sub-task 1, submitted systems were asked to predict a specific inflected form of a given lemma. In sub-task 2, systems were given a lemma and some of its specific inflected forms, and asked to complete the inflectional paradigm by predicting all of the remaining inflected forms. Both sub-tasks included high, medium, and low-resource conditions. Sub-task 1 received 24 system submissions, while sub-task 2 received 3 system submissions. Following the success of neural sequence-to-sequence models in the SIGMORPHON 2016 shared task, all but one of the submissions included a neural component. The results show that high performance can be achieved with small training datasets, so long as models have appropriate inductive bias or make use of additional unlabeled data or synthetic data. However, different biasing and data augmentation resulted in disjoint sets of inflected forms being predicted correctly, suggesting that there is room for future improvement.

Introduction

Morphology—the study of how words change form to express grammatical categories—is a fundamental component of human language. While English morphology is relatively simple (e.g., walk to walked), the majority of the world's languages employ complex morphological engines. In languages like Turkish or Spanish, a single verb can manifest in dozens of distinct forms depending on tense, person, number, and case. The CoNLL-SIGMORPHON 2017 shared task was designed to push the boundaries of automated morphological generation across a typologically diverse set of 52 languages.

Task Description

The shared task was divided into two primary challenges:

  1. Inflection (Sub-task 1): Systems were provided with a lemma and a target set of morphological features (e.g., run + Past Tense) and tasked with producing the correct inflected form (ran).
  2. Paradigm Completion (Sub-task 2): Systems were given a lemma and a partial set of its inflected forms, requiring them to predict all missing entries in the word's inflectional paradigm.
Visual representation of Sub-task 1 (Inflection) and Sub-task 2 (Paradigm Completion).

Both tasks were evaluated across three data conditions: Low (100 training examples), Medium (1,000), and High (10,000), testing the robustness of systems in data-scarce environments.

Methods

The 2017 task marked a significant shift toward neural architectures. Out of 25 unique system submissions, 24 incorporated neural components, primarily building upon the success of sequence-to-sequence (seq-to-seq) models.

  • Neural Seq-to-Seq: Most teams utilized Encoder-Decoder models that process the lemma character-by-character to generate the inflected form.
  • Hard Monotonic Attention: A key innovation involved forcing models to process strings strictly from left to right. This "hard monotonic" inductive bias mimics human morphological processes and significantly reduced "hallucinations" in low-resource settings.
  • Data Augmentation: To combat data scarcity in the 100-example condition, teams employed synthetic data generation. This included "hallucinating" new training pairs by identifying common patterns or using unlabeled Wikipedia text to teach models basic string manipulation and copying skills.

Datasets

The benchmark covered 52 languages across 10 language families. Data was primarily sourced from Wiktionary, supplemented by projects like Alexina and manual fieldwork for languages like Haida and Basque. All data followed the UniMorph schema, providing a consistent cross-linguistic framework for morphological labeling.

Examples of the UniMorph schema and data format across English (en), Spanish (es), and German (de).

Results and Discussion

The results demonstrated the clear dominance of neural models. In high-resource settings, the top-performing system achieved an average accuracy of 95.32% across all 52 languages.

The low-resource condition (100 examples) proved more challenging but highlighted the importance of inductive bias. While a rule-based baseline managed only ~38% accuracy, the best neural systems exceeded 50%, with some reaching 80% on specific languages. However, performance varied significantly by language; regular languages with simple suffixing were handled with ease, whereas morphologically complex languages like Latin remained difficult.

Conclusion

The CoNLL-SIGMORPHON 2017 shared task confirmed that neural sequence-to-sequence models are the current state-of-the-art for morphological generation. The success of these models in low-resource settings depends heavily on appropriate inductive biases and clever data augmentation strategies. Future research will likely focus on moving from orthographic (spelling-based) to phonological (sound-based) modeling and exploring cross-lingual transfer to support the world's many under-resourced languages.

Summary updated

Abstract

The CoNLL-SIGMORPHON 2017 shared task on supervised morphological generation required systems to be trained and tested in each of 52 typologically diverse languages. In sub-task 1, submitted systems were asked to predict a specific inflected form of a given lemma. In sub-task 2, systems were given a lemma and some of its specific inflected forms, and asked to complete the inflectional paradigm by predicting all of the remaining inflected forms. Both sub-tasks included high, medium, and low-resource conditions. Sub-task 1 received 24 system submissions, while sub-task 2 received 3 system submissions. Following the success of neural sequence-to-sequence models in the SIGMORPHON 2016 shared task, all but one of the submissions included a neural component. The results show that high performance can be achieved with small training datasets, so long as models have appropriate inductive bias or make use of additional unlabeled data or synthetic data. However, different biasing and data augmentation resulted in disjoint sets of inflected forms being predicted correctly, suggesting that there is room for future improvement.
BibTeX
@inproceedings{cotterell2017conll,
  author = {Ryan Cotterell and Christo Kirov and John Sylak-Glassman and Géraldine Walther and Ekaterina Vylomova and Patrick Xia and Manaal Faruqui and Sandra Kübler and David Yarowsky and Jason Eisner and Mans Hulden},
  title = {{CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages}},
  booktitle = {CoNLL Shared Task},
  year = {2017},
  url = {https://openreview.net/forum?id=kyH1YxAh-J-},
}
Figure