Abstract
This paper describes the NYU-CUBoulder systems developed for the SIGMORPHON 2020 Shared Task. We address Task 0 (supervised morphological inflection) and Task 2 (unsupervised morphological paradigm completion). Our approach models morphological inflection as a sequence-to-sequence problem using a Transformer architecture. To handle the character-level similarities between lemmas and their inflected forms, we propose a pointer-generator Transformer that explicitly allows copying input characters. Our best system for Task 0 ranked 6th out of 23, while our Task 2 system achieved 2nd place out of 7 submissions.
Introduction
Morphology, the study of how words are formed, presents unique challenges for neural models. In many languages, a single dictionary form (lemma) can generate thousands of inflected forms based on grammatical features. This paper explores how modern Transformer architectures can be adapted for these character-level transformations, particularly in low-resource settings where training data is scarce.
Problem Definition
The authors focus on two primary challenges within the SIGMORPHON 2020 framework:
- Task 0 (Supervised Inflection): Generating a specific inflected form from a lemma and a set of morphosyntactic features.
- Task 2 (Unsupervised Paradigm Completion): Discovering and generating entire word paradigms from raw text and a list of lemmas, without labeled examples.
Methodology
The core innovation is the Pointer-Generator Transformer. Unlike standard Transformers that generate from a fixed vocabulary, this model uses a "soft switch" mechanism to decide at each step whether to generate a new character or copy one directly from the input lemma. This provides a strong inductive bias for morphology, where most inflected forms (e.g., hug $\rightarrow$ hugged) retain the majority of the source characters.
To further improve performance, the team employed several data augmentation strategies:
- Morphological Reinflection: Training the model to map between different inflected forms (Form A $\rightarrow$ Form B) rather than just Lemma $\rightarrow$ Form.
- Hallucination Pretraining: For low-resource languages, 10,000 synthetic examples were generated by applying extracted prefix/suffix rules to random strings.
- Ensembling: Combining predictions from multiple model initializations via majority voting to reduce errors from random initialization.
Key Results and Insights
The pointer-generator mechanism proved particularly effective in data-constrained environments. In Task 0's low-resource settings, it outperformed the vanilla Transformer by a margin of 4.46% in accuracy. For Task 2, the system achieved the 2nd highest score among participants, demonstrating the robustness of the copy mechanism in unsupervised contexts.
Ablation studies highlighted that while the pointer mechanism is crucial when data is sparse, its benefits diminish in high-resource settings where vanilla Transformers can implicitly learn to copy. Additionally, the combination of multitask training and hallucination was vital, with hallucination alone providing a 1.85% boost for low-resource languages.
Conclusion
While the Transformer-based systems are highly competitive, they still face challenges from specialized "neural transducer" models that utilize hard monotonic attention. Future work will focus on hyperparameter optimization and improving unsupervised grammar discovery from raw text, which remains a significant hurdle for the field.
Summary updated