The UniMelb Submission to the SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection

Andreas ScherbakovACL 2020

TL;DR

This paper presents the University of Melbourne's submission to the SIGMORPHON 2020 Shared Task on typologically diverse morphological inflection. They introduce a novel data hallucination technique that significantly boosts performance on low-resource languages and a non-neural system that achieves surprisingly good results by leveraging observed inflection patterns.

What Problem Does This Paper Address?

Natural Language Processing (NLP) research has historically focused on a limited set of well-documented languages, often from the Indo-European family. This has led to a gap in our understanding of how well NLP models generalize to the vast majority of the world's languages, which are low-resource and exhibit diverse morphological structures. The SIGMORPHON 2020 Shared Task 0 aims to address this by evaluating models on a broad range of typologically diverse languages, particularly focusing on their performance in low-resource scenarios. The challenge is to build models that can accurately predict inflected word forms for languages with very little training data and diverse morphological rules, moving beyond the biases of dominant language families.

Methods

The UniMelb team submitted three systems: two neural and one non-neural.

  • Non-Neural System (flexica01): This system is based on extracting and matching inflection patterns. It first identifies maximal contiguous matches between a lemma and its inflected form, allowing for swapped orders of these matched segments. These patterns are then generalized by allowing for concrete characters. A scoring function combines the frequency of a pattern, the diversity of characters within matched segments, and the specificity of the pattern (number of concrete characters). This approach aims to capture common inflectional transformations without relying on deep learning.
  • Neural Systems (flexica02 & flexica03):
    • flexica02 (Multilingual, family-based learning): This system uses a hard monotonic attention model. A key innovation is combining languages within the same family into a single dataset, adding language and genus features. This allows the model to infer common cross-lingual inflection patterns, especially beneficial for low-resource languages within a family. They also improved the alignment process between lemma and inflected forms.
  • flexica03 (Adding Hallucinated Data): This system builds upon flexica02 but incorporates a data hallucination technique. Inspired by prior work, they generate synthetic inflected forms for a given part-of-speech (POS) using a word generator. A reverse predictor (based on flexica01) then assigns tag-lemma combinations to these hallucinated forms. This aims to augment the training data, particularly for low-resource languages, by creating more plausible inflectional examples.

Datasets

The SIGMORPHON 2020 Shared Task 0 provided datasets for 90 languages. These were divided into:

  • Development Stage: 45 languages.
  • Generalization Stage: An additional 45 languages, released shortly before the evaluation.

The dataset sizes were highly imbalanced, ranging from tens of thousands of samples in some languages to a few hundred in others. The data follows the UniMorph annotation schema and is in UTF-8 format.

Key Insights and Ideas

  • Data Hallucination is Powerful for Low-Resource Languages: The paper strongly suggests that generating synthetic, plausible inflected forms can significantly improve model performance, especially when training data is scarce. This is a crucial finding for making NLP models more equitable across languages.
  • Non-Neural Systems Can Still Be Competitive: The pattern-based non-neural system achieved over 75% accuracy on 50% of languages, demonstrating that even simpler, interpretable methods can be effective, particularly when tailored to linguistic phenomena like morphological patterns.
  • Leveraging Linguistic Families Helps: Grouping languages by family and providing family-level features to neural models allows them to learn more generalizable inflectional rules, which is a smart way to transfer knowledge in low-resource settings.
  • Indo-European Bias in Scoring: The authors note that their non-neural system's scoring formula, while effective, seems to be biased towards Indo-European languages, suggesting that more language-agnostic scoring mechanisms might be needed for truly universal performance.

Key Results

  • The data hallucination technique (flexica03) significantly improved accuracy, particularly in low-resource scenarios. Figure 2 shows substantial gains for languages like Maori, Zarma, Tajik, and Middle High/Low German.
  • The non-neural system (flexica01) achieved >75% accuracy for 50% of languages, with potential to reach >90% with further refinement.
  • The multilingual neural system (flexica02) benefited from family-based training, showing improved generalization.
  • While neural systems generally outperformed non-neural ones, the non-neural system's performance was competitive, especially considering its simplicity.

Limitations and Future Work

  • Non-Neural Scoring Bias: The scoring formula for the non-neural system was noted to not fit well for the Oto-Manguean family, indicating a need for more robust scoring mechanisms.
  • Model Size and Memory: For the non-neural system, increasing the ConcreteLetterLimit improved accuracy but led to unacceptably high memory consumption, suggesting a need for pruning or optimization techniques.
  • Future Work: The authors suggest exploring further improvements by considering correlations between inflection patterns within a lemma's paradigm (paradigm clustering) and developing more sophisticated data augmentation strategies. They also hint at the potential for ongoing pruning procedures to address memory issues.

Summary updated

BibTeX
@inproceedings{scherbakov2020unimelb,
  author = {Andreas Scherbakov},
  title = {{The UniMelb Submission to the SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure