The NYU-CUBoulder Systems for SIGMORPHON 2020 Task 0 and Task 2

Assaf Singer, Katharina Kann

TL;DR

This paper presents the NYU-CUBoulder team's systems for the SIGMORPHON 2020 shared tasks on morphological inflection and unsupervised paradigm completion. They leverage a pointer-generator transformer architecture, enhanced with multitask training and hallucination pretraining, achieving strong results, including 6th place in Task 0 and 2nd place in Task 2.

What Problem Does This Paper Address?

Morphologically rich languages express a lot of grammatical information through word forms, leading to many inflected versions of a single lemma (base word). This poses a challenge for NLP systems, as it can lead to data sparsity and requires models to accurately generate these inflected forms. The SIGMORPHON 2020 shared task specifically targets two key problems:

  • Task 0: Morphological Inflection: Given a lemma and a set of morphosyntactic features (like tense, number, person), generate the correct inflected word form. This is crucial for understanding and generating text in morphologically diverse languages.
  • Task 2: Unsupervised Morphological Paradigm Completion: Given raw text and a list of lemmas, generate the complete set of inflected forms for each lemma without explicit labels for each form. This is a more challenging, unsupervised task that aims to build paradigms from unannotated data.

Methods

The NYU-CUBoulder team developed systems for both tasks, primarily building upon the Transformer architecture.

  • Core Model: Pointer-Generator Transformer: For Task 0, they adapted the standard Transformer by incorporating a pointer-generator mechanism. This allows the model to not only generate characters from its fixed vocabulary but also to directly copy characters from the input lemma. This is particularly effective because inflected forms often share many characters with their lemmas, especially in low-resource settings. The pointer-generator acts as a switch, deciding whether to generate a new character or copy an existing one, guided by attention weights.
  • Multitask Training: To address the issue of small training sets for some languages in Task 0, they employed multitask training. This involves training the model on related tasks simultaneously, effectively increasing the amount of training data. Specifically, they created a "morphological reinflection" task, where the model learns to generate inflected forms from any form in a paradigm, not just the lemma. This helps the model better understand the relationships between different word forms within a paradigm.
  • Hallucination Pretraining: For languages with very few training instances, they used hallucination pretraining. This technique generates synthetic training data by applying known suffixation and prefixation rules to existing data, effectively "hallucinating" new examples to boost the model's initial learning.
  • Task 2 Integration: For Task 2, their approach combined a provided baseline system with their Task 0 inflection models. The baseline system first extracts potential inflected forms from raw text and generates pseudo-training data. Their inflection model then uses this data to learn and generate any missing forms in the paradigm.
  • Ensembling: To further improve performance and robustness, they used ensembling, combining the predictions of multiple models (trained with different random initializations or from different training epochs) to arrive at a final decision, often through majority voting.

Datasets

  • SIGMORPHON 2020 Task 0 Dataset: This dataset comprises 90 languages, divided into 45 development languages and 45 "surprise" languages. The languages are typologically diverse, spanning multiple families.
  • SIGMORPHON 2020 Task 2 Dataset: For this task, participants were provided with a tokenized Bible in each language and a list of lemmas. The goal was to construct complete paradigms for these lemmas.

Key Insights and Ideas

  • Copying is Key for Low-Resource Inflection: The pointer-generator mechanism, enabling direct copying of characters from the lemma, significantly boosts performance, especially for languages with limited training data. This highlights the importance of leveraging the inherent structure of word formation.
  • Multitask Learning Can Be Tricky: While multitask training is generally beneficial, the paper notes that for the SIGMORPHON 2020 task, it sometimes decreased performance. They hypothesize this might be due to the relatively large training sets provided for the shared task, making the added complexity of multitask learning less impactful or even detrimental compared to focusing on the primary task.
  • Hallucination Pretraining is Effective: For extremely low-resource scenarios, hallucination pretraining proved to be a valuable technique, improving performance on languages with fewer than 1000 training instances.
  • Unsupervised Paradigm Completion Benefits from Supervised Inflection: The success in Task 2 demonstrates that leveraging strong supervised morphological inflection models (from Task 0) as components within an unsupervised framework can yield excellent results.

Key Results

  • Task 0: The NYU-CUBoulder systems achieved 6th place out of 23 participating systems. Their best performing system was a five-model ensemble of pointer-generator transformers. They observed that pointer-generator models outperformed vanilla transformers on low-resource languages.
  • Task 2: Their system secured 2nd place out of 7 submissions. Their best performing system for Task 2 was an ensemble of vanilla transformers.
  • Low-Resource Experiment: In a simulated low-resource setting, their pointer-generator transformer achieved an average accuracy of 67.61%, outperforming a vanilla transformer (63.06%) and coming within 2.45% of a state-of-the-art neural transducer (70.06%).

Limitations and Future Work

  • Multitask Training Nuances: The paper notes that multitask training, while a common technique, didn't always improve results on this specific dataset, suggesting that its effectiveness might be context-dependent and potentially less beneficial when large training sets are already available.
  • Hyperparameter Tuning: The authors acknowledge that better hyperparameter tuning could potentially lead to even better results, especially for the low-resource experiment.
  • Future Work: The paper implicitly suggests that further exploration into the interplay between different architectural components (copy mechanism, multitask training, hallucination) and their impact across varying resource levels could be fruitful. The success of the pointer-generator architecture also points towards its continued relevance for morphological tasks.

Summary updated

Abstract

We describe the NYU-CUBoulder systems for the SIGMORPHON 2020 Task 0 on typologically diverse morphological inflection and Task 2 on unsupervised morphological paradigm completion. The former consists of generating morphological inflections from a lemma and a set of morphosyntactic features describing the target form. The latter requires generating entire paradigms for a set of given lemmas from raw text alone. We model morphological inflection as a sequence-to-sequence problem, where the input is the sequence of the lemma's characters with morphological tags, and the output is the sequence of the inflected form's characters. First, we apply a transformer model to the task. Second, as inflected forms share most characters with the lemma, we further propose a pointer-generator transformer model to allow easy copying of input characters. Our best performing system for Task 0 is placed 6th out of 23 systems. We further use our inflection systems as subcomponents of approaches for Task 2. Our best performing system for Task 2 is the 2nd best out of 7 submissions.
BibTeX
@inproceedings{singer2020nyu,
  author = {Assaf Singer and Katharina Kann},
  title = {{The NYU-CUBoulder Systems for SIGMORPHON 2020 Task 0 and Task 2}},
  booktitle = {SIGMORPHON},
  year = {2020},
  url = {https://openreview.net/forum?id=wzoUQ4MUnMU},
}
Figure