Modeling rapid language learning by distilling Bayesian priors into artificial neural networks

R. Thomas McCoy, Thomas L. Griffiths

Abstract

Humans can learn languages from remarkably little experience. Developing computational models that explain this ability has been a major challenge in cognitive science. Bayesian models that build in strong inductive biases - factors that guide generalization - have been successful at explaining how humans might generalize from few examples in controlled settings but are usually too restrictive to be tractably applied to more naturalistic data. By contrast, neural networks have flexible representations that allow them to learn well from naturalistic data but require many more examples than humans receive. We show that learning from limited naturalistic data is possible with an approach that combines the strong inductive biases of a Bayesian model with the flexible representations of a neural network. This approach works by distilling a Bayesian model's biases into a neural network. Like a Bayesian model, the resulting system can learn formal linguistic patterns from a small number of examples. Like a neural network, it can also learn aspects of English syntax from a corpus of natural language - and it outperforms a standard neural network at acquiring the linguistic phenomena of recursion and priming. Bridging the divide between Bayesian models and neural networks makes it possible to handle a broader range of learning scenarios than either approach can handle on its own.

Introduction: The Poverty of the Stimulus

A central mystery in cognitive science is how human children learn language so quickly from so little data—a puzzle often called the "Poverty of the Stimulus." Historically, computational modeling has been split into two camps. Bayesian Models use strong "inductive biases" (built-in assumptions) to generalize from just a few examples, but they are mathematically rigid and struggle to process the messy, large-scale data of natural language. Conversely, Neural Networks are incredibly flexible and excel at natural language processing but possess "weak" inductive biases, requiring millions or billions of words to learn patterns that a human might pick up in dozens.

This research seeks to bridge this divide by creating a system that possesses the strong biases of a Bayesian model but the flexible representations of a neural network. The authors propose a framework where the "gut instincts" of a Bayesian model are distilled into an artificial neural network using meta-learning.

Methods: Inductive Bias Distillation

The core contribution of this work is a three-step process called Inductive Bias Distillation. This method allows researchers to bake symbolic logic into the initial connection weights of a continuous vector-based system.

  1. Characterizing the Bias: The researchers define a Bayesian prior over formal languages using a "probabilistic language of thought." This involves a set of primitives—such as concatenation, repetition, and synchrony—that can be combined to describe various grammars.
  2. Sampling Data: This Bayesian model is used to generate 25,000 different "tasks" or artificial languages. Each task consists of a set of strings following a specific rule derived from the prior.
  3. Applying Meta-Learning: Using Model-Agnostic Meta-Learning (MAML), a "student" model (a 2-layer LSTM) is trained. Instead of learning one specific language, the LSTM learns how to learn the types of languages favored by the Bayesian prior.
The Inductive Bias Distillation process: sampling tasks from a Bayesian model to meta-train a neural network.

Experimental Setup and Datasets

To evaluate the effectiveness of this distillation, the authors tested the models across several benchmarks:

  • Formal Languages: A benchmark of 56 artificial languages used in prior Bayesian research.
  • Natural Language (CHILDES): An 8.5-million-word corpus of English speech directed at children, representing the actual linguistic input a human child receives.
  • Targeted Evaluations: The researchers utilized existing benchmarks like Zorro and BLiMP_CH, alongside a new benchmark, SCaMP (Selectional Category Minimal Pairs), designed to test semantic plausibility, recursion, and priming.

Key Results: Efficiency and Complexity

The prior-trained network demonstrated significant improvements over standard neural architectures in both data efficiency and the handling of complex linguistic structures.

Learning curves comparing the prior-trained network to standard models on formal language tasks.

On formal language tasks, the prior-trained network reached the same performance level as a standard network using roughly one-tenth of the training examples, effectively achieving 10x data efficiency. When scaled to natural language using the CHILDES corpus, the prior-trained model achieved a lower perplexity (19.67) than standard neural networks.

Furthermore, the model mastered recursion—a hallmark of human language. In tests involving nested phrases (e.g., "The book on the chair in the room..."), the prior-trained model's accuracy remained stable even at 10+ levels of recursion, whereas standard models' performance degraded significantly. It also exhibited a stronger priming effect, mirroring the human tendency to process sentences more easily if they repeat a recently encountered structure.

Discussion and Future Directions

A key insight from this work is the decoupling of representation from bias. The paper proves that symbolic representations (like trees) are not strictly necessary to have symbolic-like biases; these biases can be embedded into the weights of a neural network. Interestingly, LSTMs were found to be better "students" for this distillation than Transformers, likely because the sequential nature of LSTMs better matches the recursive primitives used in the formal language prior.

While the current distillation method shows promise, future work must address its compatibility with Transformers, the backbone of modern large language models. Additionally, this framework provides a tool for cognitive scientists to test different hypothesized human biases by observing which distilled priors produce the most human-like learning curves. Scaling these distilled biases to billions of parameters remains a significant frontier for future research.

Summary updated

Abstract

Humans can learn languages from remarkably little experience. Developing computational models that explain this ability has been a major challenge in cognitive science. Bayesian models that build in strong inductive biases - factors that guide generalization - have been successful at explaining how humans might generalize from few examples in controlled settings but are usually too restrictive to be tractably applied to more naturalistic data. By contrast, neural networks have flexible representations that allow them to learn well from naturalistic data but require many more examples than humans receive. We show that learning from limited naturalistic data is possible with an approach that combines the strong inductive biases of a Bayesian model with the flexible representations of a neural network. This approach works by distilling a Bayesian model's biases into a neural network. Like a Bayesian model, the resulting system can learn formal linguistic patterns from a small number of examples. Like a neural network, it can also learn aspects of English syntax from a corpus of natural language - and it outperforms a standard neural network at acquiring the linguistic phenomena of recursion and priming. Bridging the divide between Bayesian models and neural networks makes it possible to handle a broader range of learning scenarios than either approach can handle on its own.
BibTeX
@inproceedings{mccoy2025modeling,
  author = {R. Thomas McCoy and Thomas L. Griffiths},
  title = {{Modeling rapid language learning by distilling Bayesian priors into artificial neural networks}},
  booktitle = {Nature Communications},
  year = {2025},
  doi = {10.1038/s41467-025-59957-y},
}
Figure