Quantity doesn't buy quality syntax with neural language models

Marten van Schijndel, Aaron Mueller, Tal Linzen

Abstract

Recurrent neural networks (RNNs) can learn to predict upcoming words remarkably well on average; in syntactically complex contexts, however, they often assign unexpectedly high probabilities to ungrammatical words. We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained. We find that gains from increasing network size are minimal beyond a certain point. Likewise, expanding the training corpus yields diminishing returns; we estimate that the training corpus would need to be unrealistically large for the models to match human performance. A comparison to GPT and BERT, Transformer-based models trained on billions of words, reveals that these models perform even more poorly than our LSTMs in some constructions. Our results make the case for more data efficient architectures.

Introduction

A prevailing philosophy in deep learning, often referred to as the "Bitter Lesson," suggests that scaling up data and compute is the primary path to overcoming performance bottlenecks. In the domain of natural language processing, this has led to the development of massive models trained on trillions of tokens. However, do these models truly master the underlying structural rules of language, or are they simply better at statistical pattern matching? This paper investigates whether scaling model capacity and training data can resolve the persistent syntactic deficits observed in neural language models (LMs).

Methods

To isolate the effects of scale, we trained 125 distinct Long Short-Term Memory (LSTM) language models, systematically varying two parameters:

  1. Model Size: Hidden layer dimensions ranging from 100 to 1600 units.
  2. Training Data Size: Corpus sizes of 2M, 10M, 20M, 40M, and 80M words, drawn from disjoint subsets of the WikiText-103 corpus.

Evaluation was conducted using a targeted syntactic evaluation paradigm based on the Marvin and Linzen (2018) challenge set. Models were presented with minimal pairs of sentences (one grammatical, one ungrammatical) and were considered accurate if they assigned a higher probability to the grammatical version. We focused on subject-verb agreement and reflexive anaphora, using high-frequency verbs (is/are, was/were, has/have) to ensure even the smallest models had sufficient exposure to the target vocabulary. Finally, we compared our LSTMs against massive, publicly available Transformer models: GPT and BERT.

Key Insights: The Limits of Scaling

Our systematic evaluation revealed several critical findings regarding the relationship between scale and syntactic competence:

  • Diminishing Returns on Model Capacity: Increasing the hidden layer size improved syntactic accuracy significantly up to 400 units. However, beyond this point, further widening the network (to 800 or 1600 units) provided negligible benefits.
  • Data Scaling is Not a Silver Bullet: While increasing the training corpus from 2M to 10M words yielded clear improvements, further doublings resulted in inconsistent and marginal gains.
  • Transformers and Data Efficiency: Despite being trained on orders of magnitude more data (billions of words), GPT and BERT did not consistently outperform our LSTMs. In many syntactic conditions, they performed similarly to or worse than the LSTMs, suggesting that the Transformer architecture does not inherently solve the data efficiency problem for syntax.
  • The Complexity Paradox: Surprisingly, models often struggled more with shorter sequences, such as coordinated verb phrases ("The authors laugh and have books"), than with longer ones. This indicates that the failure is not merely a matter of memory or long-distance dependency tracking, but a fundamental inability to generalize simple syntactic patterns.
Figure 1: Performance on short VP coordination (e) shows that models struggle even with simple, short-distance dependencies.

Results and Analysis

The performance across different syntactic constructions highlights a clear divide between "easy" and "hard" syntax. Local number agreement—where the subject and verb are adjacent—is mastered quickly, with models reaching >99% accuracy with only 10M words of training data.

Figure 2: Agreement accuracy across various syntactic constructions as a function of hidden size and corpus size. Horizontal lines represent Chance, GRNN, BERT, GPT, and Human performance levels.

In contrast, more complex constructions like object relative clauses and reflexive anaphora proved much more difficult. Accuracy on object relative clauses plateaued around 80% (with a relativizer) or 70% (without one), regardless of model or data size.

By extrapolating the error reduction rate observed when doubling data from 20M to 40M words, we estimated the data requirements for human-level performance. For the most challenging constructions, our projections suggest it would take over 10 billion tokens to match humans, and trillions of tokens to reach near-perfect accuracy (99.99%). This is an impractically large amount of data for rules that human children learn from a fraction of that exposure.

Conclusion

Our findings challenge the assumption that syntactic deficits in neural LMs can be solved simply by scaling. Both network capacity and corpus size show clear diminishing returns. The fact that massive Transformers like GPT and BERT still struggle with basic agreement in certain contexts further underscores the need for a shift in research focus. Rather than relying solely on scale, the field should explore architectures with stronger structural inductive biases or training methods that incorporate explicit syntactic supervision to achieve more human-like data efficiency.

Summary updated

Abstract

Recurrent neural networks can learn to predict upcoming words remarkably well on average; in syntactically complex contexts, however, they often assign unexpectedly high probabilities to ungrammatical words. We investigate to what extent these shortcomings can be mitigated by increasing the size of the network and the corpus on which it is trained. We find that gains from increasing network size are minimal beyond a certain point. Likewise, expanding the training corpus yields diminishing returns; we estimate that the training corpus would need to be unrealistically large for the models to match human performance. A comparison to GPT and BERT, Transformer-based models trained on billions of words, reveals that these models perform even more poorly than our LSTMs in some constructions. Our results make the case for more data efficient architectures.
BibTeX
@inproceedings{schijndel2019quantity,
  author = {Marten van Schijndel and Aaron Mueller and Tal Linzen},
  title = {{Quantity doesn't buy quality syntax with neural language models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=hjV_4RIwk4},
}
Figure