Kate McCurdy, Sharon Goldwater, Adam Lopez Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020)
Introduction
For decades, cognitive scientists have debated whether the human brain processes language through explicit symbolic rules or through connectionist pattern matching. A classic battleground for this debate is morphology—specifically, how we generate plurals for novel words. While recent research suggested that modern Encoder-Decoder (ED) neural networks had achieved human-like performance on English morphology, this study argues that English is "easy mode" due to its single overwhelming majority rule (-ed for past tense, -s for plurals).
In contrast, the German plural system features five different suffixes (-e, -(e)n, -er, -s, -∅) with no single majority class. Even rare suffixes like -s can be "productive," meaning humans instinctively apply them to specific types of new words. This paper investigates whether neural models can learn these minority-class generalizations or if they simply default to the most frequent patterns encountered during training.
Methods
The authors conducted a two-part comparative study:
- Human Study (The "Wug Test"): 150 native German speakers provided plural forms and Likert-scale ratings for 24 "nonce" (invented) words. These words were categorized as "Rhymes" (phonologically similar to existing words) or "Non-Rhymes" (atypical sounds).
- Computational Modeling: A standard Encoder-Decoder model (2-layer LSTM with attention) was trained on 11,243 German noun forms from the UniMorph dataset. The model was tasked with morphological transduction: taking a singular noun and its gender as input to produce the plural form.
The model was evaluated on the same 24 nonce words, and its output probabilities were compared against human production frequencies and entropy.
Key Results
The Frequency Trap and Generalization Failure
While the model achieved high accuracy on standard held-out real German words (88.8%), it failed to replicate human behavior on novel words. Neural models proved heavily biased toward the most frequent class in the training data. For neuter nouns (the gender of the test stimuli), the model predicted the suffix -e over 80% of the time, whereas humans only used it in approximately 45% of cases.
Humans often use minority suffixes as "defaults" for phonologically atypical words (Non-Rhymes). The ED model did the opposite: it became more likely to use the frequent -e suffix for atypical words (91.7% application) rather than recognizing the "elsewhere" conditions humans utilize. For instance, while humans increased their use of the rare -s suffix for Non-Rhymes, the model's usage of -s actually decreased.
Lack of Variability and "Noise"
Human responses showed high entropy, reflecting natural linguistic variability. In contrast, the model's predictions were highly "peaked." Furthermore, when the model did not select its top choice, its lower-ranked predictions were often phonologically invalid strings ("other") rather than alternative valid German plural markers.
Discussion and Conclusion
The study demonstrates that "near-ceiling" accuracy on standard benchmarks does not necessarily imply that a model has captured the underlying cognitive mechanisms of human language. The ED model's over-reliance on high-frequency patterns and its failure to capture minority-class productivity suggest significant limitations in using current architectures as cognitive models for complex morphological systems like German.
Future work may involve training models on distributional data rather than single targets or exploring whether newer architectures like Transformers better capture these nuanced generalizations. Ultimately, the paper highlights the importance of testing models on linguistically diverse systems to truly understand their capacity for human-like generalization.
Summary updated