SIGMORPHON-UniMorph 2023 Shared Task 0: Typologically Diverse Morphological Inflection

Omer Goldman, Khuyagbaatar Batsuren, Salam Khalifa, Aryaman Arora, Garrett Nicolai, Reut Tsarfaty, Ekaterina VylomovaSIGMORPHON 2023 · OpenReview

TL;DR

This paper presents the SIGMORPHON-UniMorph 2023 shared task, which challenged systems to perform morphological inflection across 26 typologically diverse languages. The task focused on generalization to unseen lemmas and feature combinations, pushing the boundaries of current NLP models in handling complex linguistic phenomena.

What Problem Does This Paper Address?

Morphological inflection, the process of changing a word's form to indicate grammatical function (like tense, number, or case), is a fundamental aspect of human language. Building computational models that can accurately perform this task across a wide variety of languages, especially those with complex or unusual structures, remains a significant challenge. Previous shared tasks have shown that models often overfit to training data, performing poorly on unseen words or feature combinations. This year's SIGMORPHON-UniMorph task aimed to address this by:

  • Typological Diversity: Including languages from 9 different language families, covering a broad spectrum of linguistic structures.
  • Lemma-Split Data: Providing training data where inflected forms are not explicitly seen, forcing models to generalize from lemmas and feature sets.
  • Hierarchical Annotation Schema: Utilizing a new schema (UniMorph 4.0) that better accommodates complex morphological phenomena like polypersonal agreement.
  • Challenging Writing Systems: Including Japanese, with its logographic Kanji characters, to test robustness against unseen graphemes.

Methods

The shared task evaluated various approaches to morphological inflection. Participants submitted nine systems, a mix of neural and non-neural models. Key methodological innovations highlighted include:

  • Hard Attention Mechanisms: One system explored using hard attention directly over static character representations for explainability, moving beyond standard soft attention.
  • Bidirectional Decoding: Another system experimented with novel decoder architectures capable of decoding from both left-to-right and right-to-left simultaneously.
  • Non-neural System Improvements: Efforts were made to enhance existing non-neural affixing systems, demonstrating their continued relevance.
  • Ensembling: Combining multiple models, including a weighted finite-state transducer (WFST) with a neural system, proved effective.

The task also provided two baselines: a neural character-level transformer and a non-neural statistical affixing system.

Datasets

The task utilized the UniMorph 4.0 dataset, which provides lemma-split morphological paradigms for a wide range of languages. For this year's task, 26 languages were included, spanning 9 primary language families. The data was structured according to a hierarchical annotation schema. Notably, Japanese was a new addition, presenting a unique challenge due to its logographic Kanji characters.

Key Insights and Ideas

  • Neural vs. Non-Neural Performance: Overall, neural systems outperformed non-neural systems, but the non-neural baseline was surprisingly strong on languages with smaller inflection tables (like Danish and English), suggesting that generalization can be aided by simpler models when data is scarce.
  • The Challenge of Unseen Characters: Japanese, with its logographic Kanji, proved particularly difficult for neural systems. Most systems required special handling, such as replacing unseen characters with placeholders, highlighting the need for robust grapheme-level processing.
  • Orthography Matters: The study revealed that writing systems significantly impact model performance. The difference in performance between vocalized and unvocalized Arabic and Hebrew data underscored the complex interplay between orthography and morphological analysis.
  • Hierarchy vs. Flattening: Most systems opted to flatten the hierarchical feature structure, treating features as a flat list. This suggests that current models may not fully leverage explicit hierarchical linguistic information.
  • Extinct and Low-Resource Languages Remain Difficult: Languages like Ancient Greek, Sanskrit, Navajo, and Sámi continued to pose significant challenges, indicating that further research is needed for these linguistic areas.

Key Results

The submitted systems achieved impressive results on average, with the best system reaching over 99% accuracy on some languages. However, performance varied dramatically across languages, with the best system's accuracy ranging from 55.6% to 99.4%. The neural baseline system was significantly hampered by the Japanese data, but its performance improved considerably when Japanese was excluded. Ensembling proved beneficial, with one combined system achieving the best non-neural performance.

Limitations and Future Work

The authors acknowledge that despite advancements, significant gaps remain, particularly for extinct, low-resource, and logographic languages. The paper suggests that further research should focus on:

  • Closing the Performance Gap: Developing more robust models for challenging languages.
  • Orthographic Influence: Investigating the role of writing systems more deeply, especially for languages with non-alphabetic scripts.
  • Leveraging Linguistic Structure: Exploring ways to better utilize hierarchical feature structures and other linguistic knowledge.
  • Handling Unseen Characters: Developing more sophisticated mechanisms for dealing with unseen graphemes, potentially through copy mechanisms or improved character-level representations.

Summary updated

BibTeX
@inproceedings{goldman2023sigmorphon,
  author = {Omer Goldman and Khuyagbaatar Batsuren and Salam Khalifa and Aryaman Arora and Garrett Nicolai and Reut Tsarfaty and Ekaterina Vylomova},
  title = {{SIGMORPHON-UniMorph 2023 Shared Task 0: Typologically Diverse Morphological Inflection}},
  booktitle = {SIGMORPHON},
  year = {2023},
  url = {https://openreview.net/forum?id=lboRkkHj0K},
}
Figure