The NYU-CUBoulder Systems for SIGMORPHON 2020 Task 0 and Task 2

Assaf Singer, Katharina Kann

Abstract

This paper describes the NYU-CUBoulder systems developed for the SIGMORPHON 2020 Shared Task. We address Task 0 (supervised morphological inflection) and Task 2 (unsupervised morphological paradigm completion). Our approach models morphological inflection as a sequence-to-sequence problem using a Transformer architecture. To handle the character-level similarities between lemmas and their inflected forms, we propose a pointer-generator Transformer that explicitly allows copying input characters. Our best system for Task 0 ranked 6th out of 23, while our Task 2 system achieved 2nd place out of 7 submissions.

Introduction

Morphology, the study of how words are formed, presents unique challenges for neural models. In many languages, a single dictionary form (lemma) can generate thousands of inflected forms based on grammatical features. This paper explores how modern Transformer architectures can be adapted for these character-level transformations, particularly in low-resource settings where training data is scarce.

Problem Definition

The authors focus on two primary challenges within the SIGMORPHON 2020 framework:

  1. Task 0 (Supervised Inflection): Generating a specific inflected form from a lemma and a set of morphosyntactic features.
  2. Task 2 (Unsupervised Paradigm Completion): Discovering and generating entire word paradigms from raw text and a list of lemmas, without labeled examples.
Examples of morphological inflection tasks showing the mapping from lemma and features to an inflected form.

Methodology

The core innovation is the Pointer-Generator Transformer. Unlike standard Transformers that generate from a fixed vocabulary, this model uses a "soft switch" mechanism to decide at each step whether to generate a new character or copy one directly from the input lemma. This provides a strong inductive bias for morphology, where most inflected forms (e.g., hug $\rightarrow$ hugged) retain the majority of the source characters.

To further improve performance, the team employed several data augmentation strategies:

  • Morphological Reinflection: Training the model to map between different inflected forms (Form A $\rightarrow$ Form B) rather than just Lemma $\rightarrow$ Form.
  • Hallucination Pretraining: For low-resource languages, 10,000 synthetic examples were generated by applying extracted prefix/suffix rules to random strings.
  • Ensembling: Combining predictions from multiple model initializations via majority voting to reduce errors from random initialization.

Key Results and Insights

The pointer-generator mechanism proved particularly effective in data-constrained environments. In Task 0's low-resource settings, it outperformed the vanilla Transformer by a margin of 4.46% in accuracy. For Task 2, the system achieved the 2nd highest score among participants, demonstrating the robustness of the copy mechanism in unsupervised contexts.

Comparison of vanilla Transformer (Trm) vs. Pointer-Generator (Trm-PG) on Task 2, showing the performance boost from the copy mechanism.

Ablation studies highlighted that while the pointer mechanism is crucial when data is sparse, its benefits diminish in high-resource settings where vanilla Transformers can implicitly learn to copy. Additionally, the combination of multitask training and hallucination was vital, with hallucination alone providing a 1.85% boost for low-resource languages.

Conclusion

While the Transformer-based systems are highly competitive, they still face challenges from specialized "neural transducer" models that utilize hard monotonic attention. Future work will focus on hyperparameter optimization and improving unsupervised grammar discovery from raw text, which remains a significant hurdle for the field.

Summary updated

Abstract

We describe the NYU-CUBoulder systems for the SIGMORPHON 2020 Task 0 on typologically diverse morphological inflection and Task 2 on unsupervised morphological paradigm completion. The former consists of generating morphological inflections from a lemma and a set of morphosyntactic features describing the target form. The latter requires generating entire paradigms for a set of given lemmas from raw text alone. We model morphological inflection as a sequence-to-sequence problem, where the input is the sequence of the lemma's characters with morphological tags, and the output is the sequence of the inflected form's characters. First, we apply a transformer model to the task. Second, as inflected forms share most characters with the lemma, we further propose a pointer-generator transformer model to allow easy copying of input characters. Our best performing system for Task 0 is placed 6th out of 23 systems. We further use our inflection systems as subcomponents of approaches for Task 2. Our best performing system for Task 2 is the 2nd best out of 7 submissions.
BibTeX
@inproceedings{singer2020nyu,
  author = {Assaf Singer and Katharina Kann},
  title = {{The NYU-CUBoulder Systems for SIGMORPHON 2020 Task 0 and Task 2}},
  booktitle = {SIGMORPHON},
  year = {2020},
  url = {https://openreview.net/forum?id=wzoUQ4MUnMU},
}
Figure