Modern large language models (LLMs) demonstrate remarkable linguistic proficiency, often matching or exceeding human performance on standard benchmarks. However, this success relies on an unprecedented scale of data. Models like GPT-3, as detailed by Mark Chen et al. in Language Models are Few-Shot Learners, require exposure to hundreds of billions or even trillions of tokens to achieve general task competence. In stark contrast, human children acquire language fluently within their first few years of life, exposed to a mere fraction of that data—typically estimated at around 100 million words.
This massive discrepancy is known as the data efficiency gap (or sample efficiency gap). As Oana Ignat et al. point out in Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models, bridging this gap is one of the most fundamental open questions in natural language processing. It challenges the cognitive plausibility of current AI models and suggests that next-token prediction on raw text lacks the efficient learning mechanisms inherent in biological systems.
The Problem with Brute-Force Scaling
Why does this gap matter? If we can simply scale up datasets to achieve high performance, why worry about data efficiency? In How Can We Accelerate Progress Towards Human-like Linguistic Generalization?, Tal Linzen argues that high benchmark scores achieved through brute-force memorization mask a lack of true linguistic generalization. When models are evaluated on out-of-distribution tasks or restricted to human-scale data, their apparent competence often collapses.
For instance, Jared Moore notes in Language Models Understand Us, Poorly that despite consuming tens of thousands of times more data than a child, state-of-the-art models still struggle with robust compositional understanding. When training data is restricted to a developmentally realistic scale, the limitations of standard architectures become glaringly obvious.
The authors of Large Language Models and the Argument from the Poverty of the Stimulus demonstrated that LLMs trained on human-scale data fail to acquire complex syntactic rules, such as parasitic gaps and across-the-board movement. Similarly, in Do LLMs learn a true syntactic universal?, the authors found that while models can internalize universal grammatical constraints when given superhuman amounts of data, they fail to do so at human developmental scales, particularly in lower-resource languages.
This empirical reality revitalizes the classic linguistic "Poverty of the Stimulus" argument: raw text alone, at human scales, does not contain enough statistical information to unambiguously induce the hierarchical structure of language without some form of innate bias.
Why Humans Are More Efficient: Inductive Biases
To understand why the gap exists, we must look at how standard neural networks process language versus how humans do. Standard Transformers treat text as a linear sequence of tokens. They must discover through brute-force correlation that language is actually organized hierarchically into nested phrases.
Alex Warstadt and Samuel R. Bowman explored this in Can neural networks acquire a structural bias from raw linguistic data?, finding that while some hierarchical preferences emerge at scale, doing so requires significantly more data than a human child needs. Without an explicit structural prior, models waste enormous computational and data resources learning basic structural rules.
To fix this, researchers are exploring ways to bake structural awareness directly into the learning process. For example, the authors of Sneaking Syntax into Transformer Language Models with Tree Regularization introduced TREEREG, a regularization method that constrains a Transformer's internal representations to align with tree-like syntactic structures. By providing this structural guidance, the authors achieved significantly better data efficiency and out-of-distribution generalization without altering the core architecture.
Beyond Text: Grounding and Interaction
Another reason for the data efficiency gap is that text-only LLMs learn language in a vacuum. A human child learns language while simultaneously experiencing the physical world, mapping words to visual, tactile, and social contexts.
Recent work shows that incorporating these modalities drastically reduces the volume of text required for learning. In Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling, the authors developed LexiContrastive Grounding (LCG), a method that uses visual information to ground word meanings in the early layers of a language model. The authors found that visual supervision significantly improves word-learning efficiency, particularly for concrete concepts, allowing models to perform well in low-data regimes.
Furthermore, human language acquisition is an active, social process, not a passive reading exercise. The authors of Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations proposed the Trial-and-Demonstration (TnD) framework, which simulates how caregivers interactively "babysit" and correct language learners. By combining active student trials with targeted teacher demonstrations, the authors demonstrated that interactive feedback dramatically accelerates vocabulary acquisition compared to standard passive pre-training.
Similarly, the authors of A Developmental Robot Model of Early Language Acquisition showed that grounding language acquisition in a robot's internal homeostatic needs and social interactions allows for the emergence of meaning from sparse, naturalistic data.
Algorithmic Adaptations for Fast Mapping
Humans also possess the ability to perform "fast-mapping"—learning a new concept or word sense from just one or two exposures. Standard language models, by contrast, require thousands of examples to reliably integrate new vocabulary or logical constraints.
To address this, researchers are turning to meta-learning and alternative training objectives. In Rapid Word Learning Through Meta In-Context Learning, Wentao Wang developed Minnow, a meta-learning framework designed to teach language models how to learn new words from minimal contextual examples, mimicking human-like data efficiency.
In a similar vein, the authors of BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language demonstrated that by scaling down model capacity and optimizing the training strategy specifically for small-scale, child-directed speech, a model can acquire robust grammatical knowledge using 6,000x less data than standard baselines.
Conclusion
The data efficiency gap highlights a profound divergence between artificial and biological intelligence. While brute-force scaling of text corpora has yielded powerful engineering tools, it bypasses the elegant, highly efficient mechanisms of human cognitive development. Closing this gap requires moving beyond the "more data" paradigm and instead focusing on structural inductive biases, multimodal grounding, and interactive learning algorithms.