Abstract
This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding. This paradigm consists of three stages: (1) pre-training of a word prediction model on a corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set representing a classification task; (3) evaluation on a test set drawn from the same distribution as that training set. This paradigm favors simple, low-bias architectures, which, first, can be scaled to process vast amounts of data, and second, can capture the fine-grained statistical properties of a particular data set, regardless of whether those properties are likely to generalize to examples of the task outside the data set. This contrasts with humans, who learn language from several orders of magnitude less data than the systems favored by this evaluation paradigm, and generalize to new tasks in a consistent way. We advocate for supplementing or replacing PAID with paradigms that reward architectures that generalize as quickly and robustly as humans.
The PAID Paradigm and Its Discontents
In recent years, progress in Natural Language Processing (NLP) has been largely defined by the PAID paradigm: Pretraining-Agnostic Identically Distributed evaluation. Under this framework, models are pre-trained on massive, arbitrary corpora, fine-tuned on a specific task, and then evaluated on a test set drawn from the same distribution as the training data.
While this approach has led to impressive scores on benchmarks like GLUE and SuperGLUE, it creates a fundamental misalignment between current AI development and the goal of achieving human-like linguistic intelligence. The PAID paradigm rewards brute-force scaling and the exploitation of statistical shortcuts rather than the development of robust inductive biases. To move toward genuine linguistic generalization, the field must shift its focus from headline accuracy to sample efficiency and structural understanding.
The Sample Efficiency Gap
A primary critique of the current paradigm is its disregard for how much data a model requires to reach a certain level of performance. There is a staggering disparity between the linguistic input required by human children and the data consumed by modern Large Language Models (LLMs):
- Human Acquisition: A typical child may hear between 10 million and 100 million words before reaching adulthood. Some populations, such as Tsimane children in Bolivia, achieve full linguistic competence with even less input—roughly 15 times less than the average American child.
- Model Pre-training: In contrast, the scale of model training has exploded. BERT (2018) was trained on 3.3 billion words, RoBERTa (2019) on 28 billion, and T5 (2019) on a massive 130 billion words.
This 1000x difference suggests that while models like T5 achieve high scores, they are not learning language in a human-like way. Current evaluation metrics treat a model that achieves 90% accuracy using 100 billion words as "better" than a model achieving 80% accuracy with 10 million words, even though the latter may possess more scientifically interesting inductive biases.
The Problem of Statistical Shortcuts
Because the PAID paradigm relies on "Identically Distributed" test sets—random splits of the training data—models are incentivized to learn "shortcuts" or dataset artifacts. For example, in Natural Language Inference (NLI) tasks, a model might learn that the presence of the word "not" is highly correlated with the label "contradiction."
Instead of learning the underlying rules of syntax and logic, low-bias architectures like Transformers excel at picking up these fine-grained statistical quirks. When these models are tested on "out-of-distribution" or expert-designed challenge sets that lack these artifacts, their performance often collapses, revealing a lack of true linguistic generalization.
A Roadmap for a Generalization Leaderboard
To address these flaws, we propose a new framework for evaluation: the Generalization Leaderboard. This framework rests on three pillars designed to reward architectures that generalize as robustly as humans:
- Standardized Pre-training: Researchers should compare models trained on fixed, moderately sized corpora (e.g., <100M words). This levels the playing field and highlights models that can learn efficiently from limited data.
- Normative Evaluation: Rather than relying solely on random splits, evaluation should include "normative" datasets—expert-crafted minimal pairs (like those in BLiMP) designed to test specific syntactic and semantic rules.
- Test-only Benchmarks: To prevent models from learning task-specific artifacts, fine-tuning on the evaluation distribution should be prohibited. Models must instead rely on the general linguistic knowledge acquired during pre-training to solve new tasks.
Conclusion
The jump in performance from BERT to T5 was driven by massive increases in data and compute, not by fundamental architectural innovations. While scaling has its merits, it should not be confused with progress toward human-like intelligence. By moving away from the PAID paradigm and prioritizing sample efficiency and robust generalization, the NLP community can begin to develop models that truly understand the structure of language.
Summary updated