Inflecting when there's no majority: Limitations of encoder-decoder neural networks as cognitive models for German plurals

Kate McCurdy, Sharon Goldwater, Adam Lopez

Abstract

Can artificial neural networks learn to represent inflectional morphology and generalize to new words as human speakers do? Kirov and Cotterell (2018) argue that the answer is yes: modern Encoder-Decoder (ED) architectures learn human-like behavior when inflecting English verbs, such as extending the regular past tense form -(e)d to novel words. However, their work does not address the criticism raised by Marcus et al. (1995): that neural models may learn to extend not the regular, but the most frequent class -- and thus fail on tasks like German number inflection, where infrequent suffixes like -s can still be productively generalized. To investigate this question, we first collect a new dataset from German speakers (production and ratings of plural forms for novel nouns) that is designed to avoid sources of information unavailable to the ED model. The speaker data show high variability, and two suffixes evince 'regular' behavior, appearing more often with phonologically atypical inputs. Encoder-decoder models do generalize the most frequently produced plural class, but do not show human-like variability or 'regular' extension of these other plural markers. We conclude that modern neural models may still struggle with minority-class generalization.

Introduction

The debate over whether human language processing relies on symbolic rules or statistical patterns has long used the "wug test" as a primary battleground. While recent advancements in Encoder-Decoder (ED) neural networks suggest they can mimic human-like rule generalization in English, English morphology is relatively simple, dominated by a single regular class. German plurals, by contrast, feature five major suffixes (/-n, -e, -s, -er, -0/) with no clear numerical majority. Crucially, human speakers utilize the infrequent /-s/ suffix as a "default" for phonologically atypical words. This study examines whether ED models can learn such minority-class defaults or if they are bound by frequency-based patterns.

Methods

The researchers compared human performance against a standard ED model (2-layer LSTM with attention).

  1. Human Study: 150 native German speakers performed a wug test on 24 nonce nouns. These included "Rhymes" (phonologically typical) and "Non-Rhymes" (atypical). Participants provided both plural productions and acceptability ratings.
  2. Model Study: The ED model was trained on the UniMorph German dataset (11,243 nouns). To simulate a population and account for model instability, 25 versions were trained with different random seeds, and their predictions were aggregated.

Key Insights: The Frequency Trap and Model Certainty

A primary finding is the "frequency trap": neural models tend to over-rely on the most common statistical candidate. In the context of neuter, monosyllabic nouns, the /-e/ suffix is most frequent. While humans used /-e/ for approximately 45% of nonce words, the model predicted it for over 80%.

Comparison of plural suffix production for nonce words between human speakers (Study 1) and the Encoder-Decoder model (Study 2).

As illustrated in the figure above, human speakers (Study 1) exhibit a rich distribution of plural markers for any given nonce word. In contrast, the ED model (Study 2) often collapses toward a single choice, typically the most frequent /-e/ suffix (indicated in blue), failing to capture the variability inherent in human linguistic behavior.

Results and Generalization Gap

The model achieved a high accuracy of 88.8% on real-world German nouns, indicating it successfully learned the training distribution. However, its performance on generalization tasks diverged sharply from human behavior.

Human production percentages and acceptability ratings for different plural suffixes across Rhyme (R) and Non-Rhyme (NR) conditions.

As shown in the table above, human speakers maintain consistent usage of the /-e/ suffix across Rhyme and Non-Rhyme conditions (45.3% vs 44.7%). Conversely, the model's reliance on /-e/ surged to 91.7% for Non-Rhymes. Furthermore, humans increased their use of minority suffixes like /-s/ for "weird" words, a trend the model failed to replicate. The model essentially treated minority classes as exceptions to be memorized rather than productive rules.

Conclusion

The study demonstrates that while modern neural models are proficient at frequency-matching, they struggle with minority-class generalization. They tend to treat rare but productive markers as idiosyncratic exceptions. Future research should investigate whether training on distributional data or employing different architectures, such as Transformers, can better approximate the cognitive flexibility of human speakers in complex morphological systems.

Summary updated

Abstract

Can artificial neural networks learn to represent inflectional morphology and generalize to new words as human speakers do? Kirov and Cotterell (2018) argue that the answer is yes: modern Encoder-Decoder (ED) architectures learn human-like behavior when inflecting English verbs, such as extending the regular past tense form -(e)d to novel words. However, their work does not address the criticism raised by Marcus et al. (1995): that neural models may learn to extend not the regular, but the most frequent class -- and thus fail on tasks like German number inflection, where infrequent suffixes like -s can still be productively generalized. To investigate this question, we first collect a new dataset from German speakers (production and ratings of plural forms for novel nouns) that is designed to avoid sources of information unavailable to the ED model. The speaker data show high variability, and two suffixes evince 'regular' behavior, appearing more often with phonologically atypical inputs. Encoder-decoder models do generalize the most frequently produced plural class, but do not show human-like variability or 'regular' extension of these other plural markers. We conclude that modern neural models may still struggle with minority-class generalization.
BibTeX
@inproceedings{mccurdy2020inflecting,
  author = {Kate McCurdy and Sharon Goldwater and Adam Lopez},
  title = {{Inflecting when there's no majority: Limitations of encoder-decoder neural networks as cognitive models for German plurals}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=qofrSo7HLM},
}

Loading related content...

Figure