Modern Large Language Models (LLMs) are often described as "data-hungry." While a human child achieves linguistic proficiency by age ten after hearing roughly 100 million words, state-of-the-art models like GPT-4 are trained on trillions of tokens—a "superhuman" volume of data that exceeds a lifetime of human experience by several orders of magnitude.
This discrepancy creates a fundamental question for both AI and cognitive science: Is the success of LLMs due to their architecture, or simply their access to the entire internet? A growing body of research, exemplified by the The 2nd BabyLM Challenge, seeks to bridge this data efficiency gap by training models on "developmentally plausible" datasets of 10M to 100M words.
The Poverty of the Stimulus in AI
In linguistics, the "Poverty of the Stimulus" argument suggests that children do not receive enough data to learn the complex rules of grammar through statistics alone, implying they must have innate "inductive biases." Emmanuel Chemla and colleagues in Large Language Models and the Argument from the Poverty of the Stimulus demonstrate that when LLMs are restricted to human-scale data, they often fail to acquire the nuanced syntactic constraints that humans master naturally.
This suggests that current Transformer architectures lack the efficient internal priors found in biological systems. As Emmanuel Dupoux argues in Cognitive Science in the era of Artificial Intelligence, the goal is to "reverse-engineer" the infant learner by building models that can generalize from the same sparse, noisy input that children receive.
Engineering Sample Efficiency
To succeed under strict data constraints, researchers have moved away from "brute-force" scaling toward architectural and algorithmic refinements.
- Optimized Architectures: Models like BabyBERTa, developed by Philip A. Huebner and Dan Roth, show that smaller, carefully tuned models can outperform massive ones on grammatical benchmarks when data is scarce. Similarly, Andrey Kutuzov introduced LTG-BERT, proving that high-quality, curated corpora can rival massive web-scale data.
- Objective Functions: Instead of standard cross-entropy, Shaozhen Shi and Yevgen Matusevych utilized reverse KL divergence in Choosy Babies Need One Coach to induce "mode-seeking" behavior, allowing small models to focus on the most critical linguistic patterns from a teacher model.
- Multimodal Grounding: Humans don't learn language from text alone; they see and hear the world. WhisBERT and work on Visual Grounding explore whether connecting words to images or audio can provide the "environmental grounding" necessary to learn word meanings more efficiently than text-only models.
Competence vs. Plausibility
A striking finding in this field is that "better" models are not always "more human." Julius Steuer and Marius Mosbach found in Large GPT-like Models are Bad Babies that as models scale up, they gain "linguistic competence" (the ability to pass tests) but lose "psycholinguistic plausibility" (the ability to predict human reading effort).
This suggests a dissociation: scaling can mask a model's lack of true human-like processing. However, Eghbal A. Hosseini and Martin Schrimpf argue in Artificial neural network language models predict human brain responses that even with only 100M tokens, models can achieve near-maximal alignment with human brain activity, suggesting that the Transformer architecture does possess some inherent capacity for human-like representation, provided the data volume is realistic.
Why This Matters
By restricting the data budget, researchers force the model to rely on its internal mechanisms rather than memorization. This direction moves AI away from being a "statistical sponge" and toward becoming a scientific tool for understanding how humans acquire the most complex system we know: language. As Raphaël Millière notes in Language Models as Models of Language, these data-constrained models serve as "digital twins" that allow us to test long-standing linguistic hypotheses in a controlled, computational environment.