University of Illinois Submission to the SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection

Marc E. Canby, Aidana Karipbayeva, Bryan J. Lunt, Sahand Mozaffari, Charlotte R. Yoder, Julia HockenmaierACL 2020

TL;DR

This paper introduces a novel bidirectional decoding approach for morphological inflection, a task crucial for natural language processing across diverse languages. By allowing the model to consider both left-to-right and right-to-left context simultaneously during prediction, they achieve improved performance on a challenging dataset spanning 90 languages.

What Problem Does This Paper Address?

Morphological inflection, the process of changing a word's form to indicate grammatical features like tense, number, or gender, is a fundamental aspect of many languages. However, the sheer diversity in how languages achieve this—from simple suffixation to complex circumfixes—poses a significant challenge for NLP systems. Existing models, often based on unidirectional transformers, might struggle to capture the intricate phonetic and orthographic dependencies present in inflected forms, especially when these dependencies involve context from both before and after a specific point in the word. The SIGMORPHON 2020 Shared Task 0 specifically aimed to address this by presenting a broad range of languages, pushing for generalized models that aren't biased towards specific linguistic typologies.

Methods

The core innovation of this work lies in modifying the standard transformer encoder-decoder architecture to incorporate bidirectional decoding. Unlike traditional unidirectional decoders that predict tokens sequentially from left to right, this approach uses a synchronous bidirectional decoding strategy. This means the decoder simultaneously predicts the inflected form from both the left-to-right (L2R) and right-to-left (R2L) directions. At each decoding step, both directions attend to the predictions of the other, creating a richer contextual understanding.

Key aspects of their method include:

  • Synchronous Bidirectional Attention (SBAtt): A novel attention mechanism that allows the L2R and R2L decoders to attend to each other's predictions.
  • Shared Weights: The L2R and R2L decoders share all weight matrices, ensuring the model has the same number of parameters as a standard unidirectional transformer.
  • Beam Search Inference: A modified beam search algorithm tracks multiple L2R and R2L hypotheses, pairing them at each step to generate predictions. The final output is selected based on the highest probability-to-length ratio.
  • Label Smoothing: Used during training to improve model robustness.
  • Adam Optimizer and Warmup-Decay Learning Rate: Standard optimization techniques employed for training.

Datasets

The research utilizes the datasets provided for the SIGMORPHON 2020 Shared Task 0. These datasets comprise:

  • 90 languages in total.
  • 45 development languages: These were available for several months and span diverse families like Austronesian, Germanic, Niger-Congo, Oto-Manguean, and Uralic.
  • 45 surprise languages: Released only a week before the test deadline, these languages cover 16 families, many of which were not represented in the development set. This design choice encourages models that generalize well across typologically diverse languages.

Each language's dataset includes training and development files with lemmas, morphosyntactic tags (in the Unimorph Schema), and inflected forms. Test sets contain only lemmas and tags. The paper notes a significant disparity in data size across languages, with Finnish having ~100,000 examples and Tajik only 53.

Key Insights and Ideas

  • Bidirectional Decoding Benefits Inflection: The core insight is that considering context from both directions simultaneously can significantly improve morphological inflection, especially for languages with complex orthographic or phonetic dependencies.
  • Linguistic Motivation: The authors argue that phonetic conditioning of affixes (e.g., assimilation) makes bidirectional context linguistically relevant for inflection.
  • Generalization Across Typologies: The use of a diverse dataset and a generalized bidirectional approach helps overcome biases towards specific language structures.
  • L2R vs. R2L Preference Varies: The study reveals that different languages exhibit varying preferences for L2R or R2L hypotheses during inference, suggesting that the "best" direction is language-specific and not always a simple majority choice.
  • Faster Convergence: Preliminary experiments suggest that the bidirectional model might converge faster than the unidirectional baseline on some languages, despite having the same parameter count.

Key Results

The paper reports that their bidirectional transformer model (BI-TRM) generally equals or outperforms the baseline transformer (TRM) and RNN-based (MONO) models.

  • Macro-Averages: On average across language families, their model shows improvements in both accuracy and edit distance.
  • Language-Level Performance: Their model outperforms the baseline transformer on more than half of the languages in the dataset.
  • Family-Level Performance: The BI-TRM model outperforms the baseline transformer on 9 out of 18 language families for accuracy and equals it on one.
  • Specific Examples: Table 5 and 6 in the appendix show detailed results, with BI-TRM achieving high accuracy on many languages, sometimes surpassing the baselines by several percentage points in accuracy and significantly reducing edit distance. For instance, in the development set, their model outperforms the baseline on 27 languages for accuracy and 30 for edit distance.

Limitations and Future Work

The authors acknowledge several areas for future exploration:

  • Understanding Directional Preferences: Further investigation is needed to understand why some languages strongly favor L2R or R2L hypotheses. They propose correlating these preferences with linguistic features from the WALS database (e.g., whether suffixes favor L2R and prefixes favor R2L).
  • Comparing with Pure R2L: A more direct comparison with a purely R2L transformer baseline would be valuable.
  • Improving Beam Search: The authors suspect that the bidirectional beam search could be improved by pairing hypotheses differently or by leveraging overlap between L2R and R2L forms once the halfway point of the target is reached.
  • Multilingual Approaches: Currently, models are trained from scratch for each language. Future work could explore multilingual models trained on language families or across all involved languages to demonstrate broader benefits of bidirectional decoding.
  • Faster Convergence Phenomenon: Investigating why the bidirectional model appears to converge faster on some languages is a key area for future research.

Summary updated

BibTeX
@inproceedings{canby2020university,
  author = {Marc E. Canby and Aidana Karipbayeva and Bryan J. Lunt and Sahand Mozaffari and Charlotte R. Yoder and Julia Hockenmaier},
  title = {{University of Illinois Submission to the SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure