When a large language model correctly turns the singular noun cat into the plural cats, or nominalizes the adjective active into activity, it appears to possess a command of grammatical rules. But does it truly understand the underlying structural mechanics of language, or is it simply retrieving a memorized string from its billions of parameters?
Distinguishing between rote memorization and true rule internalization is one of the fundamental challenges in evaluating neural network generalization. Because standard evaluation datasets often contain significant overlap with pre-training text, models can achieve high accuracy by acting as advanced lookup tables. To expose whether a model has internalized a generative rule, researchers have turned to a classic tool from psycholinguistics: nonce word evaluation. By testing models on phonologically plausible but non-existent words, we can strip away the crutch of memorization and isolate true productive linguistic competence.
The Illusion of Competence: Memorization vs. Generalization
In standard natural language processing benchmarks, a model's ability to perform morphological transformations—such as inflecting a verb or deriving a new word class—is easily conflated with lexical recall. If a model encounters a word's root or its inflected forms during training, it does not need to understand grammar to generate the correct output; it only needs to replicate a statistical association it has already seen.
This evaluation blind spot was clearly demonstrated by Omer Goldman et al. in the context of morphological inflection. When models were evaluated on standard datasets, their performance appeared near-perfect. However, when the authors introduced a strict "lemma-split"—ensuring that the model never encountered a word's root during training—performance plummeted by an average of 30 points. This revealed that the models had not mastered the abstract rules of inflection; they had merely memorized specific word families.
A similar pattern emerges across other linguistic tasks. For instance, Masato Mita demonstrated that Grammatical Error Correction (GEC) models frequently fail to correct basic subject-verb agreement errors if the specific word pair was absent from the training data. Even when the model is familiar with the individual words, it struggles to apply the abstract rule of agreement to an unfamiliar combination.
When modern large language models (LLMs) encounter novel or unfamiliar terms, they often exhibit a "real word bias." As observed by Leonie Weissweiler et al., when ChatGPT is tasked with inflecting nonsense words, it frequently hallucinates and replaces the invented words with similar-looking real words from its training corpus. This bias demonstrates how strongly parametric memory overrides structural rule application.
The Nonce Word Solution
To bypass the interference of memorized lexical items, researchers adapt the classic "Wug Test"—originally designed by Jean Berko Gleason in 1958 to study language acquisition in children. If a child is shown an invented creature called a "wug" and told there are two of them, they will naturally say there are two "wugs." Because the child has never heard the word "wug" before, their ability to pluralize it proves they have internalized an abstract rule ($noun + -s$).
In computational linguistics, presenting models with linguistically valid but non-existent roots isolates structural reasoning from parametric memory. Because a nonce word has a frequency of zero in the pre-training corpus, the model cannot rely on a high-capacity lookup table; it must use its internal architectural mechanisms to process the form.
[ Real Word Evaluation ]
Input: "active" ------> Model Lookup Table ------> Output: "activity" (Memorized)
[ Nonce Word Evaluation ]
Input: "wug" ---------> Abstract Rule / Analogy -> Output: "wugs" (Generalized)
Cristiano Ciaccio leveraged this methodology to evaluate the lexical productivity and linguistic creativity of Transformer-based models. By requiring models to define and use neologisms and made-up words, the author isolated the models' capacity to deconstruct and apply word-formation rules independently of real-world vocabulary.
Nonce words are particularly effective at exposing the limitations of models dealing with morphologically rich or agglutinative languages, where words are constructed by stacking multiple suffixes. Mete Ismayilzada et al. utilized nonce-root morphological generalization to test LLMs in languages like Turkish and Finnish. The authors discovered a sharp performance collapse as word complexity increased, proving that while LLMs can handle simple, frequent real-world words, they lack a robust, systematic mechanism for compositional grammar when forced to operate on unfamiliar roots.
Furthermore, nonce words allow researchers to evaluate structural understanding independently of subword tokenization flaws. Modern tokenizers often split words into arbitrary statistical fragments that violate linguistic boundaries. By testing models on invented words, Thinh Hung Truong et al. demonstrated that while tokenizers frequently break affixal negations (like turning an invented root into disjointed tokens), larger language models are resilient enough to leverage their internal structural knowledge to correctly process the underlying negation rule.
How Neural Networks Generalize: Rules vs. Analogy
If a model successfully generalizes a pattern to a nonce word, how is it achieving this? In linguistics, there is a long-standing debate over whether the mind processes language using explicit, symbolic rules or through analogical generalization over stored examples (exemplars). Nonce word testing provides a window into which strategy neural networks employ.
Valentin Hofmann et al. investigated this by probing LLMs with phonologically plausible non-words to analyze English adjective nominalization. The authors found that LLMs do not utilize symbolic, algebraic rules. Instead, they rely on analogical generalization, comparing the novel input to clusters of similar real words stored in their parameters. Crucially, the authors identified a fundamental divergence from human cognition: while humans generalize based on type frequency (the number of unique word types that follow a pattern), LLMs are heavily driven by raw token frequency (how often specific words appear), leading to distinctly non-human patterns of linguistic behavior.
Nonce words also help locate exactly where this structural knowledge is stored within a network's architecture. Gal Astrach et al. used phonologically valid non-words to trace how models handle subphonemic features and long-distance dependencies like vowel harmony. Their probing experiments revealed a division of labor: local phonological rules are captured within static phoneme embeddings, whereas long-distance structural dependencies are resolved dynamically within the contextualized layers of the model's encoder.
This balance between data distribution and rule application is delicate. As shown by Brandon Prickett et al., neural networks are capable of learning minority "default" linguistic patterns, but their ability to apply these defaults to entirely novel templates or "mutant" word segments remains tightly bound to the frequency and distribution of the data they encountered during training.
Cognitive and Cross-Target Parallels
The tension between rule-based computation and lexical memorization is not unique to artificial neural networks; it directly mirrors human language learning and cognitive processing.
For example, Xiulin Yang et al. extended neural-network simulations of past-tense learning to Dutch and German, demonstrating that models can replicate human-like "U-shaped" learning curves—where a learner initially memorizes irregular forms perfectly, then over-regularizes them as they learn a general rule (e.g., saying "goed" instead of "went"), before finally mastering both. However, the authors noted that text-based models are highly sensitive to orthographic transparency (spelling consistency), which can distort simulations of human auditory language acquisition.
This parallel is also evident in speech studies of human language learners. Yunjuan He et al. used novel, non-lexical stimuli to demonstrate that non-native speakers apply complex phonological transformations, such as Mandarin tone sandhi, via an abstract computational mechanism rather than simple word-for-word memorization. Yet, this capacity varies by rule type; Yike Yang et al. found that when human learners face highly complex or variable phonological rules, they often default to a lexical memorization mechanism rather than a generalized computational rule when tested on pseudo-words.
Designing Robust Generalization Benchmarks
To accelerate progress toward human-like linguistic intelligence, the AI community must design evaluation frameworks that explicitly prevent models from exploiting statistical shortcuts. As Tal Linzen argues, standard evaluation metrics reward models for learning dataset-specific quirks. To verify true generalization, test sets must be intentionally drawn from a different distribution than the training data.
To operationalize this, researchers employ several rigorous data partitioning strategies:
- Lemma and Gloss Splits: Rather than randomly splitting data, researchers ensure that the set of word roots in the test set is entirely disjoint from the training set. This approach was used by David Guriel et al. to evaluate complex multi-argument verbal morphology in Georgian, and by Federico Tavella et al. to test whether computer vision models can recognize underlying phonological properties in novel, unseen signs in American Sign Language.
- Vocabulary Stratification: This technique separates evaluation metrics into distinct tiers based on word familiarity. As detailed by Andrés Mafla et al. in an out-of-vocabulary challenge report, tracking performance separately for seen vocabulary versus entirely novel words explicitly quantifies the performance gap caused by a model's reliance on memorized training strings.
- Controlled Data Partitioning: To prevent models from relying on surface-level heuristics, test sets can be engineered to minimize n-gram overlap with training data or split along precise linguistic axes. Alissa Ostapenko demonstrated that creating challenge splits that isolate sub-tasks (like separating speech recognition from intent classification) exposes hidden generalization failures. Similarly, Tomoki Sugimoto et al. built a controlled Japanese temporal inference dataset split by specific tense fragments, revealing that models which appear successful on standard data fail completely when forced to generalize logical temporal principles to unseen structural patterns.
By shifting away from standard accuracy metrics and embracing nonce words and strict structural splits, the field can reliably measure whether language models are truly learning the generative rules of human language or simply repeating the past.
Key Foundational Work
Contextual Information Improves OOV Detection in Speech (2010) — contributed 1 concept elements to this direction.
International Conference on Computational Linguistics COLING 1969: Preprint No. 59 (1969) — contributed 1 concept elements to this direction.