Rate estimation revisited

Samuel J. GershmanPsyArXiv 2024 · DOI

TL;DR

This paper bridges a decades-old divide in learning theory by showing that "representational" learning (estimating rates of events) can be achieved using the same error-driven machinery found in "associative" models like Rescorla-Wagner. By moving to continuous time and adopting a relative response rule, the author provides a computationally efficient way to explain complex phenomena like timescale invariance and background rate effects.

What Problem Does This Paper Address?

For over 50 years, two camps have argued over how animals learn:

  1. The Associative View: Animals learn based on contiguity (how often a stimulus and reward happen together). This is exemplified by the Rescorla-Wagner model, which uses prediction errors to update "associative strengths."
  2. The Representational View: Animals learn the structure of their environment, specifically the rates at which rewards occur. This view, exemplified by Rate Estimation Theory (RET), argues that animals compare the reward rate during a stimulus to the background reward rate.

While RET explains certain behaviors better—such as why learning speed depends on the ratio of time intervals rather than absolute time (timescale invariance)—its original mathematical formulation is computationally "clunky." It requires solving linear systems that scale poorly with the number of stimuli and lacks a natural way to handle uncertainty. This paper seeks to fix RET’s math while showing it isn't as different from associative models as previously thought.

Methods

The author proposes a new algorithmic version of Rate Estimation Theory. Key features include:

  • Continuous-Time Error-Driven Learning: Instead of discrete time steps, the model uses a continuous-time update rule. It calculates a "global prediction error"—the difference between the actual reward and the sum of estimated rates of all present stimuli.
  • Recursive Estimation: The paper derives a recursive Maximum Likelihood Estimation (MLE) update. Surprisingly, this update looks almost exactly like the Rescorla-Wagner rule, but it updates rate estimates rather than abstract associative strengths.
  • Bayesian Integration: To handle the "initialization" of rates and the animal's uncertainty, the author incorporates a Bayesian framework using a Gamma distribution prior. This allows the model to be sensitive to "sample size" (how long the animal has observed the environment).
  • Relative Response Rule: Unlike associative models that "read out" strength directly, this model generates a response only when the information gain (the ratio of the stimulus-plus-background rate to the background rate alone) exceeds a threshold.

Datasets

The paper validates the model using several classic and recent experimental datasets from Pavlovian conditioning literature:

  • Gibbon et al. (1977) & Gibbon & Balsam (1981): Classic pigeon autoshaping experiments that established the law of timescale invariance.
  • Balsam et al. (2024): Recent large-scale data on how learning depends on temporal relationships.
  • Harris & Gallistel (2024): Data used to compare different mathematical formulations of the "informativeness" of a stimulus.

Key Insights and Ideas

  • The Learning vs. Performance Distinction: The "aha moment" is that the learning rule (how we update internal values) can be associative/error-driven, while the performance rule (how we decide to act) is representational/comparative.
  • Rescorla-Wagner is a Rate Estimator: The paper shows that the classic Rescorla-Wagner model is essentially trying to estimate reinforcement rates, but it gets "tripped up" by discrete time bins.
  • Timescale Invariance as an Analytical Result: By combining the new learning rule with an information-theoretic decision rule, the author analytically derives the empirical law that reinforcements to acquisition are inversely proportional to the stimulus's "informativeness."
  • Dopamine Compatibility: Because the new model uses error-driven learning, it provides a theoretical bridge to neurobiology, suggesting that the brain's dopamine system (known for reporting prediction errors) could be the engine for rate estimation.

Key Results

  • Simulation Success: The model accurately recovers true reward rates in simulations and replicates the "learning curves" seen in animal experiments.
  • Superior Model Fit: Using the Bayesian Information Criterion (BIC), the author compared two versions of the timescale invariance law. The model's derived "C/T" law (Cycle time / Trial time) decisively outperformed the alternative "I/T" law (Intertrial interval / Trial time) across three major datasets, with posterior probabilities exceeding 0.99.
  • Computational Efficiency: The new algorithm operates in real-time with space and time complexity that scales linearly with the number of stimuli, solving the quadratic complexity issues of the original RET.

Limitations and Future Work

  • Background Rate Assumptions: The derivation of timescale invariance still relies on an assumption about how animals perceive the background rate ($1/RI$), which isn't derived from first principles within the model itself.
  • Update Timing: The model assumes updating happens continuously or at regular intervals. The author notes we lack strong empirical data on exactly when the brain triggers these updates (e.g., at every reward, or every few milliseconds?).
  • Complex Environments: While the model handles multiple stimuli, further work is needed to see how it performs in more complex, non-Poisson environments where reward rates change dynamically over time.

Summary updated

Abstract

Two views of Pavlovian conditioning have dominated theoretical discourse. The classical associative view holds that associations are learned based on temporal contiguity between stimuli, and conditioned responses directly reflect associative strength. The representational view, exemplified by Rate Estimation Theory (Gallistel & Gibbon, 2000), holds that animals learn the structure of the stimulus distribution, from which a measure of contingency between stimuli is derived and used to generate conditioned responses. Unlike contiguity, contingency is a relative measure, comparing the rate of reinforcement in the presence of a stimulus to the background rate. This turns out to be crucial for explaining the effects of manipulating the background rate while holding the stimulus-conditional rate constant (i.e., changing contingency without changing contiguity). It has also been argued that contiguity theories face irremediable conceptual difficulties stemming from the coercion of continuous time into discrete bins. This paper makes two contributions to the debate. First, it shows that Rate Estimation Theory faces its own computational and conceptual problems. Second, it shows how to fix these problems while retaining the core of the theory. Surprisingly, this leads to the insight that rates can be estimated using an algorithm closely resembling a classical associative theory (the Rescorla-Wagner model). The key difference lies in the response rule rather than in the learning rule.
BibTeX
@misc{gershman2024rate,
  author = {Samuel J. Gershman},
  title = {{Rate estimation revisited}},
  year = {2024},
  doi = {10.31234/osf.io/47q5y},
  howpublished = {PsyArXiv},
}

Loading related content...

Figure