Universal linguistic inductive biases via meta-learning

R. Thomas McCoy, Erin Grant, Paul Smolensky, Thomas L. Griffiths, Tal Linzen

TL;DR

Researchers have developed a framework using meta-learning to bake specific linguistic inductive biases—the "innate" rules humans might use to learn language—into neural networks. By finding a parameter initialization that facilitates rapid learning across a space of synthetic languages, they demonstrate that neural models can inherit human-like constraints, such as a preference for specific syllable structures and the ability to generalize from minimal data.

What Problem Does This Paper Address?

Human children are remarkably efficient language learners, acquiring complex grammars from relatively sparse and noisy data—a phenomenon known as the "poverty of the stimulus." Linguists argue this is possible because humans possess inductive biases: innate pressures or constraints that guide how we generalize.

In contrast, standard neural networks are "blank slates." While flexible, they typically require massive datasets to learn linguistic patterns and often generalize in ways that are decidedly non-human. The challenge is that these models are "bottom-up" and data-driven, making it difficult to manually "program" in the specific linguistic constraints found in symbolic theories like Optimality Theory (OT). This paper bridges that gap, providing a way to give neural networks controllable, theory-driven biases so researchers can study their effects on acquisition.

Methods

The authors use Model-Agnostic Meta-Learning (MAML) to find a neural network initialization that encodes universal linguistic principles. Their approach follows a three-step pipeline:

  1. Define the Bias Space: They adopt the framework of Optimality Theory, where "universal" factors are a set of constraints (e.g., "syllables should start with a consonant"), and "non-universal" factors are the specific rankings of those constraints in a given language.
  2. Meta-Training: They generate a distribution of thousands of synthetic languages based on these OT principles. A sequence-to-sequence LSTM model is then "meta-trained" to become a fast learner. Instead of learning one language, the model learns how to learn any language in this distribution using only a few examples (100-shot learning).
  3. Verification: The authors then test the resulting "initial state" to see if it actually contains the intended biases. They do this by measuring "ease of learning" (how many examples are needed to reach 95% accuracy) and performing "poverty of the stimulus" experiments.
Summary of the three-step approach to imparting inductive biases.

Datasets

The study uses synthetic datasets based on a simplified version of syllable structure typology:

  • Meta-training set: 20,000 unique synthetic languages, each with its own constraint ranking and phoneme inventory.
  • Meta-test set: 1,000 held-out languages to evaluate 100-shot learning performance.
  • Analysis datasets: Specifically designed "unnatural" languages (e.g., those using constraints like NOONSET, which are not found in the training distribution) to test the strength and specificity of the imparted biases.

Key Insights and Ideas

  • Biases as Initializations: The core conceptual breakthrough is treating "Universal Grammar" as a parameter initialization ($M_0$). Universal factors are baked into the starting weights, while language-specific factors are learned during the fine-tuning process.
  • Soft vs. Hard Constraints: Unlike symbolic models that might find certain languages "unlearnable," this meta-learned model has "soft" biases. It can learn typologically rare or "unnatural" languages, but it takes significantly more data to do so. This aligns well with human experimental results.
  • Implicational Universals: The model meta-learns that certain patterns imply others. For example, if a language fixes complex syllable structures in one way, the model "expects" it to fix simpler structures in a consistent manner, even if it hasn't seen those simpler cases yet.
  • Emergent Categories: Without explicit labels, the model learns to treat phonemes as functional classes (consonants vs. vowels) because that distinction is a consistent "universal" across all meta-training languages.

Key Results

  • Learning Efficiency: The meta-learned model achieved 98.8% accuracy on new languages after just 100 examples, compared to only 6.5% for a randomly initialized baseline.
  • Typological Preference: Languages that followed the "natural" constraints (ONSET/NOCODA) were 5.6 times easier to learn than those following "unnatural" constraints (Figure 6a).
  • Generalization: In "poverty of the stimulus" tests, the model successfully generalized to entirely new phonemes and longer word lengths, whereas the baseline model failed completely (Figure 7).
Results showing the meta-trained model generalizes better to novel example types than the baseline.

Limitations and Future Work

  • Length Constraints: The model's bias for generalizing to longer strings was restricted to the lengths it encountered during meta-training. It struggled to generalize from length 5 to length 6.
  • Synthetic vs. Natural Data: This study used synthetic languages to ensure the "ground truth" biases were known. The authors suggest that future work could meta-learn directly from natural language data to discover which biases are actually present in human speech (Figure 8).
  • Scaling: While successful for syllable structure, scaling this to complex syntax or semantics remains a significant next step for the field.
Proposed future direction: meta-learning from natural languages to discover inherent biases.

Summary updated

Abstract

How do learners acquire languages from the limited data available to them? This process must involve some inductive biases - factors that affect how a learner generalizes - but it is unclear which inductive biases can explain observed patterns in language acquisition. To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases. This framework disentangles universal inductive biases, which are encoded in the initial values of a neural network's parameters, from non-universal factors, which the neural network must learn from data in a given language. The initial state that encodes the inductive biases is found with meta-learning, a technique through which a model discovers how to acquire new languages more easily via exposure to many possible languages. By controlling the properties of the languages that are used during meta-learning, we can control the inductive biases that meta-learning imparts. We demonstrate this framework with a case study based on syllable structure. First, we specify the inductive biases that we intend to give our model, and then we translate those inductive biases into a space of languages from which a model can meta-learn. Finally, using existing analysis techniques, we verify that our approach has imparted the linguistic inductive biases that it was intended to impart.
BibTeX
@inproceedings{mccoy2020universal,
  author = {R. Thomas McCoy and Erin Grant and Paul Smolensky and Thomas L. Griffiths and Tal Linzen},
  title = {{Universal linguistic inductive biases via meta-learning}},
  booktitle = {CogSci},
  year = {2020},
  url = {https://openreview.net/forum?id=E4qAe5iDnB},
}
Figure