Training Strategies for Neural Multilingual Morphological Inflection

Adam Ek, Jean-Philippe BernardyACL 2021

TL;DR

This paper explores how different training strategies can improve neural models for morphological inflection across many languages. By experimenting with curriculum learning, multi-task learning, and language-wise label smoothing, the authors demonstrate significant gains in accuracy, particularly for low-resource languages, while also showing that their model can learn meaningful cross-lingual similarities.

What Problem Does This Paper Address?

Morphological inflection, the process of changing a word's form to convey grammatical information like tense or person, is a fundamental aspect of language. However, building neural models that can handle this task across many languages is challenging due to several factors:

  • Data Scarcity: Many languages have limited digital resources, making it hard for models to learn effectively.
  • Typological Diversity: Languages employ vastly different strategies for inflection, ranging from adding suffixes (agglutinative) to using separate words (isolating).
  • Script Differences: Languages use diverse writing systems (Latin, Cyrillic, Arabic, etc.), which can hinder knowledge transfer between them.

Existing models often struggle to generalize across these variations. This paper aims to improve multilingual inflection models not by changing the core architecture, but by focusing on novel training strategies to better handle these challenges.

Methods

The authors employ an LSTM encoder-decoder architecture with attention as their base model, similar to existing work. Their key contributions lie in the novel training strategies they investigate:

  • Curriculum Learning: They implement a competence-based curriculum where the model is gradually exposed to more difficult examples based on its estimated ability. This aims to help the model transfer knowledge from easier examples to harder ones.
  • Multi-task Learning: Beyond predicting the inflected word character by character, the model also learns to predict the Levenshtein operations (insertion, deletion, substitution, copy) required to transform the lemma into its inflected form. This abstraction from script is hypothesized to improve generalization.
  • Language-wise Label Smoothing: Instead of uniformly smoothing the loss across all incorrect characters, this strategy smooths the loss based on the target language's alphabet. The idea is that predicting a character from the correct script is "less wrong" than predicting one from a completely different script.
  • Scheduled Sampling: This technique addresses the "exposure bias" problem where models trained only on gold data struggle at inference time. It involves probabilistically using the model's own predictions as input during training, gradually decreasing the reliance on gold data.

Datasets

The study uses data covering 38 languages of varying typology, genealogy, and scripts. For low-resource languages (defined as having fewer than 10,000 training examples), the authors augment the original dataset with "hallucinated data" generated using a technique inspired by Anastasopoulos and Neubig (2019). This hallucination process involves replacing random subsequences in existing words with sampled n-grams to create new, synthetic examples, aiming to retain some orthographic information.

Key Insights and Ideas

  • Training Strategies Matter: The paper strongly suggests that how a model is trained can be as crucial as its architecture, especially for complex multilingual tasks.
  • Abstraction Aids Generalization: The multi-task learning approach, by predicting Levenshtein operations, shows promise in abstracting away from specific scripts, potentially improving cross-lingual transfer.
  • Script is a Significant Hurdle: The model's performance is notably impacted by script differences, as seen in the clustering of language embeddings and the difficulty with languages like Classical Syriac.
  • Low-Resource Languages Benefit Most: Strategies like curriculum learning and language-wise label smoothing appear to provide the most significant improvements for languages with less training data.

Key Results

The paper reports that their multilingual model, enhanced with these training strategies, achieves strong performance. For languages with over 30,000 examples, accuracy reaches around 98%. For languages with around 10,000 natural examples and no hallucinated data, accuracy drops to about 50%. Languages that received hallucinated data show improved performance, with exact match accuracy often above 50%, though this varies significantly by language. The authors also note that scheduled sampling, surprisingly, degraded overall performance by 3.04 percentage points, particularly affecting low-resource languages, which they hypothesize benefits from gold data input.

Limitations and Future Work

The authors acknowledge that languages using different scripts remain a significant challenge. They suggest transliteration as a potential solution for future work. They also plan to explore curriculum learning in more detail, potentially estimating model competence based on batch accuracy rather than just training steps. Another avenue is to combine model loss with insights from language acquisition research for example scoring. Finally, they intend to further investigate language-wise label smoothing, particularly how the smoothing parameter should be tuned based on the number of characters and languages.

Summary updated

BibTeX
@inproceedings{ek2021training,
  author = {Adam Ek and Jean-Philippe Bernardy},
  title = {{Training Strategies for Neural Multilingual Morphological Inflection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure