Abstract
Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, Mans Hulden. Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection.
2017.
Introduction
In many languages, a single "dictionary" word (lemma) can take dozens or even hundreds of different forms depending on its role in a sentence—a process called inflection. While humans handle this naturally, it's a significant hurdle for Human Language Technology (HLT) like machine translation or speech recognition, especially for "morphologically rich" languages like Turkish or Finnish.
The 2017 CoNLL-SIGMORPHON shared task established a massive benchmark for morphological generation across 52 typologically diverse languages. The task focused on morphological reinflection: generating a specific word form given a lemma and a set of features (e.g., run + [Verb, Past Tense] → ran). Systems were evaluated across 52 different languages with varying rules (prefixes, suffixes, vowel shifts) and under different data constraints, from "high-resource" (10,000 examples) down to "low-resource" (only 100 examples).
Methods
The shared task was organized into two primary tracks:
- Sub-Task 1 (Inflected Form from Lemma): Predicting a single form given a lemma and a feature bundle.
- Sub-Task 2 (Paradigm Completion): Filling in the missing entries of a word's full inflectional table (paradigm) given only a few observed forms.
The most successful systems utilized Neural Sequence-to-Sequence (seq2seq) models, typically RNNs with attention mechanisms. However, standard seq2seq models often fail with small datasets. To combat this, top-performing teams introduced several innovations:
- Hard Monotonic Attention: This forces the model to move strictly left-to-right, mirroring how most morphology works, unlike standard "soft" attention.
- Data Augmentation: Teams "hallucinated" new training data by identifying common string transformations and applying them to random strings or unlabeled Wikipedia text.
- Inductive Biases: Explicitly biasing models toward "copying" the input string, which is a safe default for many morphological changes.
Datasets
The shared task utilized a massive, standardized dataset covering 52 languages from 10 different language families. Most data (47 languages) was extracted and cleaned from Wiktionary using the UniMorph schema, a cross-linguistically consistent feature set. Additional data for endangered or under-resourced languages like Haida and Khaling was provided by field linguists.
Training sets were provided in three sizes: 100 (Low), 1,000 (Medium), and 10,000 (High) examples per language.
Key Results and Insights
The 2017 task marked a major shift as nearly every submission was neural, confirming that neural models have officially surpassed traditional finite-state methods in this domain.
- Performance: Top systems achieved an average of 95.32% accuracy in high-resource settings. Even in low-resource settings (100 examples), the best systems reached ~50% accuracy, a remarkable feat considering the complexity of the task.
- Winning System: The CLUZH team (University of Zurich) was the overall winner, utilizing a neural parameterization of traditional monotonic transducers.
- Generalization: In Basque, models achieved 85% accuracy even on feature combinations never seen during training, proving that neural models can decompose and recombine complex feature bundles correctly.
- Synthetic Data: One of the most effective strategies was creating "silver-standard" data. By applying observed rules to random strings, models learned the process of inflection rather than just memorizing specific words.
Conclusion
Despite the success, several challenges remain. Models operate on characters (spelling), but morphology is often driven by sound (phonology). Systems that don't account for phonological rules may struggle with irregular spellings. Furthermore, simply using raw Wikipedia text didn't significantly improve performance, suggesting a need for better semi-supervised learning methods. Future work should also investigate Active Learning to identify "diagnostic" forms that reveal the most about a language's rules.
Summary updated