Abstract: Antonios Anastasopoulos, Graham Neubig. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Morphological inflection—the process of transforming a base word (lemma) into a specific grammatical form (e.g., changing the Spanish aguar to aguà for the 2nd person plural present indicative)—is a fundamental task in natural language processing. While neural models excel when training data is abundant, they struggle with the "long tail" of the world's 7,000+ languages. In low-resource settings with as few as 100 examples, standard models often succumb to label bias, memorizing specific character sequences rather than learning generalizable morphological rules. This paper introduces a novel neural architecture and training strategy that improves accuracy in these settings by 15 percentage points.
Proposed Methods
The authors propose a multi-faceted approach to overcome data scarcity, focusing on architectural disentanglement and robust data augmentation.
Two-Step Attention Decoder
Standard sequence-to-sequence models typically concatenate morphological tags and lemma characters into a single input string. This paper introduces a decoder that separates these inputs. As shown in the architecture diagram below, the model first attends to the morphological tags to establish the grammatical intent (the "what"). It then uses this context to attend to the lemma characters to determine the specific transformation (the "how"). This disentanglement prevents the model from being overwhelmed by character-level noise in the lemma when trying to identify the required grammatical change.
Data Hallucination
To combat label bias, the researchers developed a "hallucination" technique to expand the training set. By aligning lemmata with their inflected forms, the model identifies the "stem" (the invariant part of the word). The system then generates 10,000 synthetic examples by replacing the internal characters of the stem with random characters from the language's alphabet while keeping the prefixes and suffixes intact. This forces the model to focus on the process of inflection rather than memorizing specific real-world stems.
Three-Phase Training Schedule
The model is trained using a structured three-phase approach:
- Warm-up: The model learns a simple copying task to master character-level output generation.
- Cross-lingual Training: The model is trained on a combination of the target low-resource language, a related high-resource language, and the hallucinated data.
- Fine-tuning: A final pass is conducted on the target language data using scheduled sampling to increase robustness against its own prediction errors.
Experimental Results and Insights
Using the SIGMORPHON 2019 shared task datasets (covering 100 language pairs), the proposed model achieved a macro-averaged accuracy of 63.8%, significantly outperforming the previous state-of-the-art baseline of 48.5%.
Key insights from the study include:
- Hallucination is Powerful: The most significant performance gain (nearly 12 percentage points) was attributed to data hallucination. Even without cross-lingual transfer, hallucination alone allowed the model to reach 60.1% accuracy.
- The Script Barrier: While cross-lingual transfer is effective, its utility is limited by orthography. If the transfer language and target language use different scripts (e.g., Latin vs. Arabic), the model fails to find a common representation, highlighting a need for phonological mapping in future work.
- Interpretability: Attention visualizations confirmed that the model correctly identifies specific tags (like "plural") to trigger the generation of corresponding suffixes.
Conclusion and Future Work
By combining a disentangled attention mechanism with strategic data hallucination, this work pushes the boundaries of what is possible in low-resource NLP. Future research will address the "script barrier" and explore how these techniques might be adapted for "root-and-pattern" morphology, such as that found in Semitic languages, where inflectional changes are non-contiguous.
Summary updated