Bayesian theories of conditioning in a changing world

Aaron Courville, Nathaniel D. Daw, David S. TouretzkyTrends 2006 · DOI

TL;DR

This paper proposes a Bayesian framework for understanding animal conditioning, suggesting that "surprise" signals a changing environment, which in turn accelerates learning. By modeling the world as dynamic, Bayesian agents can better explain why animals adapt their learning rates in response to unexpected events, offering a more principled account than traditional associative learning theories.

What Problem Does This Paper Address?

Classical conditioning, a fundamental area of behavioral science, has long been explained by associative learning models. However, these models often struggle to account for how animals learn faster when faced with surprising events – situations where the expected outcome of a stimulus doesn't match reality. This paper tackles this gap by re-examining this "surprise" phenomenon through the lens of modern Bayesian inference, proposing that it's not just a quirk, but a crucial signal of environmental change that drives learning. The authors aim to bridge the gap between traditional behavioral psychology and computational approaches to cognition by showing how Bayesian principles can offer a more unified and powerful explanation for these learning phenomena.

Methods

The authors frame animal conditioning within a Bayesian inference framework. They propose a "generative" world model where animals learn to predict not just the occurrence of reinforcers (like food) but also the underlying "latent causes" that generate observable stimuli and outcomes. Crucially, this model assumes that the parameters governing these relationships can change over time.

  • Bayesian Inference: The core method involves using Bayes' rule to update beliefs about the world model's parameters based on observed data (stimulus-reinforcer pairings).
  • Latent Variable Model: A specific generative model is discussed (Box 2), where unobservable "latent causes" explain the co-occurrence of stimuli and reinforcers. This allows for a richer representation of the environment than purely discriminative models that only predict outcomes.
  • Modeling Change: The key innovation is modeling the rate of change of the world model's parameters. When events are surprising, it signals a higher rate of change, leading to increased uncertainty about the current state of the world.
  • Associability as Uncertainty: The paper draws a parallel between the "associability" concept in associative learning (Pearce-Hall model) and Bayesian uncertainty. Higher uncertainty (due to perceived change) leads to greater influence of new evidence, thus speeding up learning.

Datasets

The paper does not introduce new datasets. Instead, it reinterprets and analyzes existing experimental results from the animal conditioning literature. The "datasets" are therefore the experimental protocols and behavioral outcomes described in the cited studies (e.g., latent inhibition, blocking, overshadowing experiments).

Key Insights and Ideas

  • Surprise as a Signal of Change: The central idea is that surprising events are not just errors, but valuable signals that the underlying rules of the environment have changed. This realization is key to accelerating learning.
  • Bayesian Uncertainty Drives Learning Speed: In a Bayesian framework, uncertainty about the world model's parameters is directly linked to learning speed. Higher uncertainty, triggered by surprise, makes the agent more receptive to new information.
  • Generative vs. Discriminative Models: The authors argue that phenomena involving surprising neutral stimuli (not just reinforcers) suggest that animals employ generative models, which attempt to understand the full pattern of events, rather than just discriminative models that solely predict outcomes.
  • The Pearce-Hall Model as a Precursor: The paper shows how the Pearce-Hall model's concept of "associability" can be elegantly reinterpreted within a Bayesian framework as a measure of uncertainty driven by perceived environmental change.
  • Partial Reinforcement Extinction Effect (PREE) Explained: The PREE, where partially reinforced stimuli extinguish slower, is explained by the Bayesian model's prediction that detecting a decrease in reinforcement probability is harder against a noisy background, leading to slower inference of change and thus slower extinction.

Key Results

The paper presents a theoretical framework and uses it to reinterpret existing experimental findings. While not reporting new quantitative results, it offers qualitative explanations for several phenomena:

  • Accelerated Learning: The Bayesian model explains why surprising reinforcers (e.g., unblocking experiments) and surprising neutral stimuli (e.g., overshadowing counteracting latent inhibition) lead to faster learning.
  • Reversal of Effects: The model accounts for how surprise can reverse previously established learning deficits, such as blocking.
  • PREE Explained: The model provides a statistical explanation for the slower extinction of partially reinforced stimuli, contrasting with predictions from simpler associative models.
  • Support for Generative Models: The influence of surprising neutral stimuli on learning is presented as evidence favoring generative over discriminative Bayesian models of conditioning.

Limitations and Future Work

  • Computational Intractability: The authors acknowledge that exact Bayesian inference can be computationally intractable. They raise the question of what approximations animals might use (Box 3).
  • Generalization of Surprise: It's unclear how learning about change generalizes between stimuli or contexts (Box 3).
  • Neurophysiological Mechanisms: The paper notes the suggestion of specialized brain mechanisms for detecting abrupt changes and asks how these relate to behavioral inference (Box 3).
  • Human Learning Parallels: The authors suggest investigating whether similar phenomena involving surprise and change are observed in human learning experiments (Box 3).
  • Stochasticity Nuances: While the PREE is explained, the paper notes that the precise role of stochasticity in uncertainty and associability requires further experimental investigation.

Summary updated

BibTeX
@inproceedings{courville2006bayesian,
  author = {Aaron Courville and Nathaniel D. Daw and David S. Touretzky},
  title = {{Bayesian theories of conditioning in a changing world}},
  booktitle = {Trends},
  year = {2006},
  doi = {10.1016/j.tics.2006.05.004},
}
Figure