Abstract: Marten van Schijndel, Aaron Mueller, Tal Linzen. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Introduction
Scaling up neural language models (LMs) by adding more parameters and training data yields diminishing returns for syntactic processing. This paper demonstrates that even massive models like GPT and BERT struggle with complex grammar, suggesting that reaching human-level syntactic competence requires better architectural inductive biases rather than just "more of everything."
While modern language models are incredibly good at predicting the next word on average (low perplexity), they often stumble on the "long tail" of complex linguistic structures. A classic test of this is subject-verb agreement: in a sentence like "The keys to the cabinet are/is on the table," a model must ignore the "distractor" noun (cabinet) and link the verb (are) back to the head subject (keys*).
Previous research showed that LSTMs often fail these tests. The authors ask a fundamental question: Is this a capacity problem? If we simply train larger models on more data—following the "Bitter Lesson" philosophy that scale eventually solves all architectural hurdles—will these syntactic deficits disappear?
Methodology
The researchers conducted a systematic scaling study by training 125 different LSTM language models. They varied two primary dimensions:
- Model Capacity: Hidden layer sizes ranging from 100 to 1600 units.
- Data Quantity: Training sets ranging from 2 million to 80 million words (subsets of WikiText-103).
To ensure the results weren't just quirks of LSTMs, they compared their models against GPT and BERT (Transformer-based models trained on billions of words). They used the Marvin and Linzen (2018) challenge set, which consists of "minimal pairs" across various syntactic constructions, including prepositional phrases, object relative clauses, and VP coordination.
Key Insights and Results
The 400-Unit Ceiling and Data Plateaus
Increasing the hidden layer size improved performance up to 400 units, but gains vanished beyond that. Adding more "brain power" to the RNN didn't help it understand hierarchy any better. Similarly, while more data helped initially, the returns diminished rapidly. Extrapolating the learning curves, the authors estimated that reaching human-level performance would require tens of billions of tokens, and near-perfect accuracy would require trillions—orders of magnitude more data than a human child needs to master the same structures.
Transformers Aren't a Silver Bullet
Surprisingly, GPT and BERT—despite their massive scale—performed as well as or worse than the smaller LSTMs in 7 out of 10 syntactic conditions. This suggests the struggle with syntax is an architectural trait shared by both RNNs and Transformers. As shown in the figure above, in tricky cases like object relative clauses, accuracy leveled off around 70-80%, far below human performance (indicated by the dashed red line).
Non-Linear Generalization
Models struggled to generalize from short patterns to long ones, but also failed in the opposite direction. For example, they were often better at agreement in long coordinated phrases than in short ones, defying the standard assumption that "longer is harder."
Conclusion
The study concludes that "quantity doesn't buy quality." Increasing the corpus size from 10M to 80M tokens often resulted in negligible or inconsistent accuracy gains for the most difficult syntactic structures. Instead of just throwing more GPUs at the problem, the field should investigate stronger inductive biases (architectures that inherently understand nested hierarchies) or external supervision signals to help models learn the underlying rules of language more efficiently.
Summary updated