Abstract
Tal Linzen. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
2020.
Introduction
In recent years, the Natural Language Processing (NLP) community has witnessed a "gold rush" of high scores on benchmarks like GLUE and SuperGLUE. Models such as T5 have reached performance levels that appear to rival human accuracy. However, Tal Linzen (2020) argues that these metrics are deceptive. This "near-human" performance is achieved through an evaluation paradigm that prioritizes massive data consumption over the robust, sample-efficient linguistic generalization characteristic of human intelligence.
The Problem: The PAID Paradigm
Linzen identifies the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm as a primary obstacle to progress. In this framework, models are permitted unlimited pretraining data and are subsequently evaluated on test sets that share the same statistical distribution as their training data.
This setup encourages models to exploit dataset-specific "shortcuts"—such as associating the word "not" with contradiction—rather than learning the underlying structural rules of language. Because the test sets are drawn from the same distribution as the training sets, models that capture fine-grained statistical noise are rewarded over models that attempt to learn robust linguistic rules.
The Data Disparity
A central critique in Linzen’s work is the staggering difference in data requirements between AI and humans. While a model like T5 is trained on approximately 130 billion words, a typical human child is exposed to fewer than 100 million words before reaching adulthood.
This 1000-fold difference suggests that current architectures are fundamentally inefficient. Linzen highlights the Tsimane community in Bolivia, where children learn language with significantly less verbal interaction than children in Western societies, as a benchmark for true linguistic efficiency. If our AI cannot learn from such limited data, it has not truly "solved" language acquisition.
Failure of Generalization
Despite high benchmark scores, models often fail on simple logic and syntax. For instance, a model fine-tuned on Natural Language Inference (NLI) might correctly identify a contradiction in one context but fail to understand that "the detective followed the suspect" does not entail "the suspect followed the detective."
This indicates that models are performing sophisticated pattern matching on statistical artifacts rather than genuine linguistic reasoning. Current models lack the inductive biases—the innate building blocks—that allow humans to generalize from a few examples to a wide array of new contexts.
Proposed Framework for Evaluation
To move toward human-like AI, Linzen proposes a shift in how we measure success:
- Standardized Pretraining Scales: Instead of "more is better," we should evaluate models on their ability to learn from human-scale data (e.g., 100 million words or less).
- Normative Evaluation: We should move away from noisy, crowdsourced test sets toward expert-designed "minimal pairs" (like the BLiMP benchmark) that isolate specific linguistic principles like syntax and logic.
- The Generalization Leaderboard: A new evaluation platform that rewards models for learning quickly (sample efficiency) and performing well on out-of-distribution tasks.
Conclusion
Scaling data and compute—a process Linzen describes as "hill climbing" on metrics—has yielded impressive results, but it does not necessarily bring us closer to human-like AI. While multimodal grounding is a common suggestion for improvement, the massive disparity in text volume alone suggests that our current architectures are fundamentally lacking. By shifting our focus toward inductive bias and data efficiency, we can develop models that generalize in a way that is more robust, interpretable, and human-like.
Summary updated