Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models

Ling Liu, Mans HuldenACL 2022 · DOI

Abstract

Despite the remarkable success of Transformers in Natural Language Processing, they often struggle with fundamental linguistic generalization. This paper investigates the "Wug Test"—the ability to inflect unfamiliar words—within the context of neural morphological inflection. We demonstrate that state-of-the-art models fail significantly when tested on lemmata not seen during training. We propose that this failure stems from a lack of inherent "copying bias" and introduce a novel data augmentation method using syllable-like n-gram substrings. Our results show that this method significantly improves the model's ability to generalize morphological rules to novel words across diverse language families.

Introduction: The Neural Wug Test

In 1958, Jean Berko introduced the Wug Test, a landmark experiment in linguistics demonstrating that children possess an internal grammar allowing them to apply rules to nonsense words (e.g., "one wug, two wugs"). In modern NLP, the task of Morphological Inflection—transforming a base lemma (like run) into a specific form (like running)—is often considered solved, with Transformers achieving near-perfect accuracy.

However, this performance is deceptive. Most standard benchmarks include significant "lemma overlap," where the model sees the same word stems during both training and testing. When these models are subjected to a true Wug Test—inflecting a lemma they have never encountered—their performance collapses. For instance, Finnish accuracy drops from nearly 94% in standard tasks to a mere 5.7% when the lemma is unseen.

Methodology: Inducing Copying Bias

The primary challenge is that Transformers lack an innate copying bias. In most morphological systems, the stem of a word is preserved (copied) while prefixes or suffixes are modified. Standard models attempt to learn the entire mapping from scratch rather than recognizing the stability of the stem.

To address this, we propose a Data Hallucination strategy. By augmenting the training set with synthetic examples, we can "trick" the model into learning that copying is a valid and frequent operation.

Overview of the proposed data hallucination methods: (a) generating dummy lemmata and (b) generating dummy stems to induce copying bias.

Our core innovation is the +hall-2k-substr method. Unlike previous approaches that used random character noise, we sample n-grams (2, 3, and 4-character sequences) from the actual language to create "syllable-like" synthetic words. This ensures the synthetic data matches the statistical structure of the real language, making the learned copying bias more transferable.

Experimental Results

We evaluated our approach on two distinct groups of languages: six typologically diverse languages from the CoNLL-SIGMORPHON 2018 shared task (Czech, Finnish, German, Russian, Spanish, and Turkish) and six low-resource Niger-Congo languages (Akan, Ga, Lingala, Nyanja, Southern Sotho, and Swahili). We specifically re-split these datasets to ensure zero lemma overlap between training and testing, creating a true Wug Test environment.

Performance comparison across 12 languages on the 'Wug Test'. Orange bars represent standard training (with overlap), blue bars represent the baseline on unseen lemmata, and brown bars show the dramatic improvement using substring hallucination.

The results demonstrate a massive "generalization gap" in standard models and the effectiveness of our solution:

  • Baseline Failure: In the Wug Test environment, the standard Transformer performed poorly, with Czech accuracy at 16.2% and Finnish at 5.7%.
  • Hallucination Success: With substring hallucination, Czech performance jumped to 54.2% and Finnish reached 68.5%.
  • Niger-Congo Regularity: In languages with highly regular patterns like Ga, our method achieved a perfect 100% accuracy on unseen words, up from 49.2%.

Conclusion and Future Directions

Our findings suggest that "copying" is a learned skill for neural models, not an emergent property. While our substring hallucination method significantly closes the generalization gap, challenges remain for highly irregular languages and those with discontinuous stems.

We argue that the NLP community should shift its evaluation standards. Reporting "unique lemma counts" rather than just total example counts is essential to accurately measure a model's true capacity for linguistic generalization. Only then can we determine if a model has truly passed the Wug Test.

Summary updated

BibTeX
@inproceedings{liu2022can,
  author = {Ling Liu and Mans Hulden},
  title = {{Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.84},
}
Figure