The rat as particle filter

Nathaniel D. Daw, Aaron C. CourvilleNeurIPS 2007 · OpenReview

While group-averaged learning curves often look like smooth, rational Bayesian updates, individual animals actually learn in abrupt, unstable jumps. This paper proposes a novel resolution to this contradiction: individuals act as "particle filters" using only a single sample (or hypothesis) at a time. This framework explains the noisy, erratic data of individual subjects while demonstrating how sophisticated Bayesian behavior emerges naturally when averaging across a population.

The Problem: Smooth Averages vs. Abrupt Individuals

In the study of Pavlovian conditioning—how animals learn to associate a stimulus like a bell with a reward like food—there is a glaring contradiction between theoretical models and empirical reality.

Standard Bayesian models, such as the Kalman filter, suggest that learning should be a smooth, gradual process of reducing uncertainty as more data arrives. When researchers look at data averaged across dozens of subjects, this is exactly what they see: a beautiful, smooth "learning curve." However, recent re-analyses of individual animal records, such as pigeon pecking data, reveal that individuals do not learn this way at all. Their behavior is "all-or-nothing." They suddenly start responding, and their performance fluctuates wildly from trial to trial.

Smooth group averages mask abrupt, step-like learning in individual subjects.

The challenge for theorists is to find a model that is "dumb" enough to match the erratic, step-like behavior of a single pigeon, but "smart" enough to explain why the whole flock, on average, looks like a sophisticated Bayesian calculator capable of complex cognitive feats.

The Single-Particle Model

The authors bridge this gap using Sequential Monte Carlo methods, specifically particle filtering.

  1. Conditioning as Filtering: They start with a standard generative model where stimuli produce rewards based on hidden "weights." A perfect Bayesian tracks the mean and the full covariance (uncertainty) of these weights.
  2. The Single-Particle Limit: The authors propose that an individual subject does not track a full probability distribution. Instead, they represent their belief as a single "particle"—a point estimate of the weights.
  3. The Diffusion-Jump Model: To allow the model to "learn," they use a generative model assuming that weights usually drift slightly (diffusion) but occasionally reset entirely (jumps).
  4. Sampling as Learning: On each trial, the subject samples a new hypothesis. If the current hypothesis predicts the reward poorly, the "jump" mechanism makes it highly likely the subject will ditch their current belief and sample a brand-new one from the prior.

In standard particle filters, a system keeps many particles and "resamples" them based on their weights. A single-particle filter cannot do this. However, by assuming the world is "jumpy," the individual is forced to resample from scratch when their current belief fails, mimicking the effect of a more complex filter over time.

Emergent Sophistication and Backward Blocking

The most profound insight of this paper is that the ensemble is the distribution. Even if every single pigeon is absolutely certain of a (potentially wrong) hypothesis, the diversity of hypotheses across the flock accurately represents the Bayesian posterior.

This emergent sophistication is best demonstrated in complex tasks like backward blocking. In this paradigm, an animal learns that a compound stimulus ($AB$) predicts a reward. Later, it learns that $B$ alone predicts the reward. A rational Bayesian must "retrospectively" realize that $A$ might not be necessary, requiring the tracking of complex correlations (covariance) between stimuli.

Backward blocking emerges in the ensemble. The particle filter ensemble (bottom) approximates the exact Bayesian posterior (top).

The paper shows that backward blocking can actually emerge in a population where no single individual tracks any uncertainty at all. The correlation is stored in the distribution of beliefs across the subjects. In simulations, when $B$ was shown to be a reliable predictor on its own, the average belief in $A$ dropped across the ensemble, even though $A$ was never shown again. This happened because individuals holding a "high $A$, low $B$" belief were more likely to "jump" to a new hypothesis during the $B$-only phase.

Limitations and Future Directions

While elegant, the single-particle model is an extreme proof-of-concept. Real animals likely use a small number of particles or a hybrid of sampling and exact inference. A strict single-particle filter is "brittle"—if it samples a very bad hypothesis, it might take a while to recover. Future work could explore how "multi-particle" models balance individual stability with group-level flexibility.

Furthermore, the model assumes animals have fixed expectations about how noisy the world is and how often "jumps" occur, whereas real animals likely learn these hyperparameters. Ultimately, the authors note a desperate need for more trial-by-trial individual data in the literature, as historical reliance on group averages has hidden the very dynamics this paper successfully models.

Summary updated

Abstract

Although theorists have interpreted classical conditioning as a laboratory model of Bayesian belief updating, a recent reanalysis showed that the key features that theoretical models capture about learning are artifacts of averaging over subjects. Rather than learning smoothly to asymptote (reflecting, according to Bayesian models, the gradual tradeoff from prior to posterior as data accumulate), subjects learn suddenly and their predictions fluctuate perpetually. We suggest that abrupt and unstable learning can be modeled by assuming subjects are conducting in- ference using sequential Monte Carlo sampling with a small number of samples — one, in our simulations. Ensemble behavior resembles exact Bayesian models since, as in particle filters, it averages over many samples. Further, the model is capable of exhibiting sophisticated behaviors like retrospective revaluation at the ensemble level, even given minimally sophisticated individuals that do not track uncertainty in their beliefs over trials.
BibTeX
@inproceedings{daw2007rat,
  author = {Nathaniel D. Daw and Aaron C. Courville},
  title = {{The rat as particle filter}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2007},
  url = {https://openreview.net/forum?id=SkWDmw-OZS},
}
Figure