Introduction
Neural language models (LMs) like BERT and GPT-2 exhibit impressive grammatical capabilities. However, their performance is not uniform; they often judge some sentences as more grammatical than others, even when the underlying rules are identical. This study investigates whether this variation is systematic and if it stems from word frequency in training data—a common assumption in human language acquisition.
Methodology
The researchers utilized a minimal pair methodology, comparing model probabilities for grammatically correct versus incorrect sentence pairs (e.g., "The cat walks" vs. "*The cat walk").
The evaluation focused on:
- Subject-Verb Agreement (SVA) and Reflexive Anaphora (RA).
- Noun-Centric Scoring: Performance was calculated for individual nouns across 500 templates.
- Few-Shot Learning: Models were fine-tuned on five example sentences containing novel "nonsense" words (e.g., wug) to assess how quickly grammatical properties are acquired.
The study analyzed pre-trained versions of Transformer-XL, GPT-2, and BERT, with frequency counts derived from their respective training corpora (WikiText-103, WebText, etc.).
The Frequency Paradox
The most significant finding is the near-total lack of correlation between a noun's frequency in the training corpus and the model's grammatical performance with that noun. Whether a noun appears 100 or 1,000,000 times, the model's ability to handle its agreement rules remains essentially unchanged.
Statistically, frequency explains less than 0.1% of the variation in performance. This suggests that the "more data is better" mantra does not apply to the mastery of specific lexical grammatical properties once a basic threshold is met.
Systematic Difficulty Across Architectures
Interestingly, noun performance is highly correlated across different tasks and model architectures. If GPT-2 struggles with the grammar of a specific noun, BERT and Transformer-XL are likely to struggle as well.
This consistency suggests that certain nouns possess inherent properties—perhaps semantic or structural—that make them "grammatically slippery" for current neural network architectures, regardless of the training objective or data scale.
Learnability and Few-Shot Performance
Despite these persistent "blind spots," the study found that grammar is remarkably "cheap" to learn. In few-shot experiments, models reached performance levels on novel words similar to real-world items after seeing just five examples. Furthermore, models demonstrated "semantic bootstrapping," inferring a noun's number (singular/plural) from semantic context (e.g., "The wug worked all alone") rather than just explicit grammatical markers.
Conclusion
This research challenges the assumption that exposure frequency is the primary driver of grammatical mastery in LMs. While models exhibit systematic difficulties with certain words, these properties are easily learned from minimal data. The central mystery remains: why do models continue to struggle with certain high-frequency words in their original training sets if the rules are so easily acquired? Future work is needed to identify the true source of this performance variation.
Summary updated