TL;DR
This paper investigates "data hallucination," a common augmentation technique for training NLP models on low-resource languages. They find that while it improves generalization to unseen words, it can negatively impact performance on familiar words by introducing a bias towards simple affixation, hindering the model's ability to learn complex morphological phenomena like reduplication.
What Problem Does This Paper Address?
Training NLP models for low-resource and endangered languages is challenging due to limited data. Data augmentation, particularly "data hallucination," is a popular strategy to artificially expand datasets. However, the why behind its effectiveness and its potential drawbacks remain poorly understood. This paper specifically examines how data hallucination affects neural morphological inflection models, focusing on the Gitksan language, and aims to understand its impact on generalization and the learning of different morphological processes.
Methods
The researchers trained two Transformer models for morphological reinflection: a "standard" model trained on the original dataset and an "augmented" model trained on the original data plus 10,000 hallucinated examples. They used a "one-source" model approach, where each known form in a paradigm is used to predict missing forms. A key evaluation strategy was a "wug test" scenario, where models are tested on completely unseen lexemes, and a "partially observed" test set. They also employed a controlled error analysis, focusing on suppletion and reduplication, to understand the models' decision-making processes. The MAX strategy was used to select the most confident prediction.
Datasets
The study uses a dataset of Gitksan inflection tables, programmatically extracted from an interlinear-glossed corpus. This dataset contains 1055 inflection tables with 2125 inflected forms. The data was processed to create one-to-one paradigms (one form per morphosyntactic description) and then split into four subsets: training (I_train), observed test (I_obs), development (I_dev), and a "wug test" set (I_wug) for completely unseen lexemes.
Key Insights and Ideas
- Data hallucination introduces a strong inductive bias towards concatenative morphology. The augmented model prefers simple affixation (e.g., adding "diit") over more complex transformations, which aligns well with Gitksan's dominant morphology.
- This bias is a double-edged sword. While beneficial for generalizing to unseen lexemes (the "wug test" scenario), it significantly hinders the model's ability to learn non-concatenative phenomena like reduplication and can lead to errors in processes like lenition.
- Unaugmented models tend to "memorize" rather than generalize. The standard model shows a tendency to learn specific inflectional forms, which explains its poor performance on unseen lexemes but better performance on partially observed ones.
- "Right for the wrong reasons" is a significant issue. The standard model can achieve good performance by memorizing patterns, but this doesn't reflect a true understanding of the language's morphological structure.
Key Results
- The augmented model reduced the performance gap between familiar and unfamiliar lexemes by 10% in the "wug test" scenario.
- For suppletive forms, the augmented model achieved a weighted Average Precision of .89, significantly outperforming the unaugmented model's .52, indicating it better learned the underlying process rather than just memorizing.
- However, the augmented model's accuracy on reduplication dropped from .92 (standard model) to a mere .25.
- The augmented model showed a significant drop in performance on familiar lexemes compared to the standard model, particularly when dealing with non-concatenative morphology.
Limitations and Future Work
The authors acknowledge that their study was conducted on a single language (Gitksan). They suggest that while data hallucination can be beneficial, it needs to be applied with caution. Future work should focus on developing data augmentation techniques that are more sensitive to the target language's specific morphological phenomena, potentially incorporating linguistic knowledge. Further qualitative analysis and more varied evaluation conditions are also recommended for testing automatic inflection systems.
Summary updated