Abstract
Humans can learn languages from remarkably little experience. Developing computational models that explain this ability has been a major challenge in cognitive science. Existing approaches have been successful at explaining how humans generalize rapidly in controlled settings but are usually too restrictive to tractably handle naturalistic data. We show that learning from limited naturalistic data is possible with an approach that bridges the divide between two popular modeling traditions: Bayesian models and neural networks. This approach distills a Bayesian model's inductive biases—the factors that guide generalization—into a neural network that has flexible representations. Like a Bayesian model, the resulting system can learn formal linguistic patterns from limited data. Like a neural network, it can also learn aspects of English syntax from naturally-occurring sentences. Thus, this model provides a single system that can learn rapidly and can handle naturalistic data.
Introduction: The Poverty of the Stimulus
A central question in cognitive science, often called "Plato’s Problem" or the "poverty of the stimulus," asks how human children learn the immense complexity of language from relatively little data. Historically, computational modeling has been split into two camps. Bayesian models possess "strong inductive biases"—pre-existing structured hypotheses about how rules work—making them data-efficient but often unable to handle the noise of naturalistic language. Conversely, standard artificial neural networks are highly flexible and excel at processing massive datasets but have "weak inductive biases," requiring millions of examples to learn patterns humans grasp quickly.
This research seeks to bridge this divide by creating a system that possesses the structured inductive biases of a Bayesian model while retaining the flexible, vector-based representations of a neural network.
Methodology: Inductive Bias Distillation
The authors introduce a three-step process called Inductive Bias Distillation to transfer knowledge from a symbolic prior to a connectionist architecture.
The framework consists of:
- Characterizing the Bias: A Bayesian model defines a "prior" over possible languages using primitives such as concatenation, recursion, and synchrony.
- Sampling Data: The Bayesian model generates thousands of unique "synthetic languages," each serving as a distinct learning task.
- Meta-Learning: Using Model-Agnostic Meta-Learning (MAML), a neural network (specifically an LSTM) is trained on these synthetic languages. The goal is to find an initialization that allows the network to adapt to any new language in the distribution with minimal data.
By the end of this process, the Bayesian prior is "distilled" into the starting weights of the neural network, allowing it to "learn how to learn" according to the specified biases.
Results
The distilled model was evaluated across formal rule-based languages and naturalistic English data.
Formal Language Learning
On a set of 56 abstract formal languages, the prior-trained network reached high accuracy with roughly 10 times fewer examples than a standard neural network. This performance matched the efficiency of symbolic Bayesian models while maintaining the flexibility of a neural architecture.
Natural Language Processing
When trained on the CHILDES corpus—an 8.6-million-word dataset of English spoken to children—the prior-trained model achieved a lower perplexity (19.66) than standard networks (19.75). Furthermore, the model demonstrated a superior ability to handle deep recursion (e.g., multiple levels of clausal nesting), a task where standard networks typically struggle as complexity increases.
Discussion
The most profound insight of this work is the decoupling of inductive bias from representation. It demonstrates that "strong" biases do not require symbolic, code-like representations; they can be embedded into the continuous numerical weights of a neural network. This provides a bridge between David Marr’s computational level (the what and why) and the algorithmic level (the how).
While the benefits were most pronounced when test data matched the meta-training distribution, the statistically significant improvements in natural language tasks suggest a promising path forward. Future research will focus on scaling these distillation techniques to Transformer architectures and identifying the optimal set of universal linguistic primitives.
Summary updated