Acquisition in Autoshaping

Sham Kakade, Peter DayanNeurIPS 1999 · OpenReview

While animals are remarkably adept at learning which stimuli predict rewards, standard mathematical models of conditioning often fail to predict the actual speed of this learning by orders of magnitude. This paper introduces a "Competitive Model" that treats stimuli as independent "experts" whose predictions are weighted by their reliability, successfully matching quantitative experimental data where previous theories fell short.

The Autoshaping Problem

The paper focuses on autoshaping, a simple form of classical conditioning. For example, a pigeon sees a light (the stimulus) for $T$ seconds, which is always followed by food. Eventually, the pigeon learns to peck the light in anticipation.

Decades of experiments show a striking quantitative rule: the number of trials it takes for an animal to "get it" (acquisition) depends almost entirely on the ratio between the inter-trial interval ($I$) and the duration of the stimulus ($T$). Specifically, the faster the light is presented relative to the wait time between lights ($I/T$), the faster the animal learns.

Experimental paradigm and the relationship between acquisition time and the I/T ratio.

Existing models like the famous Rescorla-Wagner model or Rate Estimation Theory (RET) struggle to explain this pacing. RET, for instance, predicts that animals should learn thousands of times faster than they actually do. It suggests an animal should be certain of a reward association after just a few trials, whereas real animals take hundreds. The authors set out to build a model that is statistically sound but matches the "slow" reality of biological learning.

A Competitive Mixture of Experts

To resolve this discrepancy, the authors propose a Competitive Model based on two main components:

  1. Kalman Filters for Rate Prediction: Instead of just tracking "associative strength," the model uses Kalman filters to estimate the rate of reward for the background context ($w_c$) and the light ($w_l$) independently. They incorporate Scalar Expectancy Theory (SET), which dictates that the uncertainty (standard deviation) of an animal's estimate stays proportional to the mean of that estimate.
  2. Cooperative Mixture of Experts: The model treats the context and the light as two "experts" making separate predictions. These predictions are combined using a weighting factor ($\pi$) called reliability.
  3. If the background context is already a "reliable" predictor of the occasional reward, it "blocks" the light from being seen as a useful predictor initially.
  4. Learning is modeled not as the discovery of the reward rate (which happens quickly), but as the slow increase in the reliability assigned to the new stimulus (the light).

Key Insights and Results

The paper relies on a meta-analysis of quantitative behavioral data from over 12 different experimental laboratories involving pigeons and rats. By evaluating this data through the lens of their new model, the authors uncover several key insights:

  • The Reliability Bottleneck: The "aha moment" is that animals aren't slow because they are bad at math; they are slow because they are cautious about changing which "expert" they trust. The actual reward rate is established quickly in the brain, but the importance (reliability) shifted to that rate grows slowly.
  • Rational Uncertainty: By constraining the model so that uncertainty never drops below a certain fraction of the mean (the SET constraint), the authors prevent the model from becoming "too certain, too fast," which was the fatal flaw of previous Bayesian approaches.
  • Matching the Empirical Law: The model successfully replicates the experimental observation that the number of rewards to acquisition ($n$) follows the formula $n \approx 300 * T/I$.
  • Invariance to Partial Reinforcement: The model correctly predicts that "partial reinforcement" (only giving food on a fraction, $1/S$, of trials) doesn't change the number of reinforcements needed to learn, even though it increases the total number of trials.

Context Blocking (US-Preexposure Effect)

The background environment is always "on." If an animal has spent a lot of time getting rewards in a specific cage, it views the cage itself as a reliable predictor. When a new light appears, the "Cage Expert" initially outvotes the "Light Expert."

Effect of prior context reinforcements on the speed of acquisition.

The model quantitatively accounts for why prior experience in a context (without the light) slows down learning once the light is introduced. As prior context reinforcements increase, the animal requires significantly more reinforcements to acquire the new association with the light.

Limitations and Future Work

While highly successful at matching empirical data, the model has a few limitations. The authors acknowledge that their specific formula for how reliability changes over time is a heuristic chosen to fit the data, lacking a "first-principles" statistical derivation. Additionally, the current model only evaluates one stimulus (the light) against the context; it does not yet address "overshadowing," where two different distinct stimuli might compete to predict the same food. Finally, while the model can theoretically account for extinction (learning that the light no longer predicts food), detailed quantitative fits for extinction were not explored in this paper.

Summary updated

Abstract

Quantitative data on the speed with which animals acquire behav(cid:173) ioral responses during classical conditioning experiments should provide strong constraints on models of learning. However, most models have simply ignored these data; the few that have attempt(cid:173) ed to address them have failed by at least an order of magnitude. We discuss key data on the speed of acquisition, and show how to account for them using a statistically sound model of learning, in which differential reliabilities of stimuli playa crucial role.
BibTeX
@inproceedings{kakade1999acquisition,
  author = {Sham Kakade and Peter Dayan},
  title = {{Acquisition in Autoshaping}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1999},
  url = {https://openreview.net/forum?id=r1NzcO-_-B},
}
Figure