Abstract
Recent years have seen exceptional strides in the task of automatic morphological inflection generation. However, for a long tail of languages the necessary resources are hard to come by, and state-of-the-art neural methods that work well under higher resource settings perform poorly in the face of a paucity of data. In response, we propose a battery of improvements that greatly improve performance under such low-resource conditions. First, we present a novel two-step attention architecture for the inflection decoder. In addition, we investigate the effects of cross-lingual transfer from single and multiple languages, as well as monolingual data hallucination. The macro-averaged accuracy of our models outperforms the state-of-the-art by 15 percentage points. Also, we identify the crucial factors for success with cross-lingual transfer for morphological inflection: typological similarity and a common representation across languages.
Introduction
Morphological inflection—the process of generating a word form from a lemma and grammatical features (e.g., walk + [verb, past tense] → walked)—is a fundamental task in NLP. While neural sequence-to-sequence models have mastered this in high-resource settings, they struggle significantly with the "long tail" of languages where training data is scarce, often limited to fewer than 100 examples. This work addresses the challenge of low-resource morphological inflection by introducing architectural innovations and data augmentation strategies.
Methodology
Two-Step Attention Architecture
Standard sequence-to-sequence models typically concatenate grammatical tags and lemma characters into a single input stream. We propose a "disentangled" approach where the decoder first attends to the grammatical tags to establish a "tag-informed" state. This state then guides the attention over the lemma characters. This separation mimics the linguistic intuition that grammar provides the structural template (the "recipe"), while the lemma provides the lexical content (the "ingredients").
Data Hallucination
To mitigate label bias in tiny datasets, we employ a data hallucination technique. By aligning lemmata and inflected forms, we identify the invariant "stem" and the variant affixes. We then generate thousands of synthetic examples by replacing the internal characters of the stem with random characters from the alphabet while preserving the prefix/suffix transformations. This forces the model to learn the rules of inflection rather than memorizing specific vocabulary.
Training Schedule
Our training pipeline consists of three phases:
- Warm-up: A simple "copy" task to establish a monotonic attention baseline.
- Cross-lingual Training: Training on a mixture of high-resource transfer data, low-resource target data, and hallucinated data.
- Fine-tuning: Refinement on the target language using scheduled sampling to improve robustness to prediction errors.
Results and Discussion
Our approach was evaluated on the SIGMORPHON 2019 shared task datasets, covering 100 language pairs across 43 distinct low-resource languages.
The results demonstrate a substantial improvement over previous benchmarks. The addition of hallucinated data alone provided a massive boost, often outperforming complex cross-lingual transfer from related languages. However, we observed a "script bottleneck": cross-lingual transfer is highly effective between genetically related languages using the same script but fails when writing systems diverge (e.g., Latin to Arabic script).
Conclusion
By combining a novel two-step attention mechanism with strategic data hallucination and a multi-phase training schedule, we pushed the state-of-the-art for low-resource morphological inflection by over 15 percentage points. Our findings highlight that structural disentanglement and synthetic data are powerful tools for overcoming data scarcity in the linguistic long tail.
Summary updated