Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time

Ian Cone, Claudia Clopath, Harel Z. ShouvalNature Communications 2024 · DOI

TL;DR

For decades, the "Temporal Difference" (TD) learning algorithm has been the gold-standard model for how the brain learns from rewards. This paper introduces FLEX (Flexibly Learned Errors in Expected Reward), a new framework arguing that the brain doesn't have a pre-set internal clock for every stimulus; instead, it must learn to represent time and reward simultaneously through plastic neural connections.

What Problem Does This Paper Address?

In reinforcement learning (RL), timing is everything. If a bell rings and food arrives ten seconds later, your brain must bridge that temporal gap. The dominant theory, TD learning, suggests the brain uses "reward prediction errors" (RPEs)—signaled by dopamine—to update its expectations.

However, TD learning relies on a "fixed temporal basis." This assumes that every time a stimulus occurs, a pre-configured chain of internal "clocks" (microstates) starts ticking. The authors argue this is biologically implausible: the brain cannot possibly maintain a dedicated, pre-wired clock for every sight, sound, and smell it might encounter. Furthermore, TD learning makes specific predictions—such as a "moving bump" of dopamine activity shifting backward from the reward to the cue—that are often absent in recent experimental data.

Methods

The authors propose FLEX, a biophysically plausible model that replaces fixed clocks with plastic representations of time.

Key components of their implementation include:

  • Core Neural Architectures (CNAs): Located in the prefrontal cortex (PFC), these consist of "Timers" (neurons that learn to sustain activity over a delay) and "Messengers" (neurons that learn to fire at the exact moment a reward is expected).
  • Dopaminergic Two-Trace Learning (dTTL): Instead of the standard TD update rule, FLEX uses a Hebbian learning rule with two competing eligibility traces (one for potentiation, one for depression). These traces are modulated by dopamine levels in the Ventral Tegmental Area (VTA).
  • Spiking Neural Network: Unlike many abstract RL models, this is a "bottom-up" simulation using integrate-and-fire neurons, making it more comparable to real electrophysiological data.

Datasets

The paper is primarily a theoretical and computational study, but it validates the FLEX model by reanalyzing several landmark experimental datasets:

  • Coddington and Dudman (2018): Electrophysiological recordings of VTA/SNc dopamine neurons.
  • Amo et al. (2022) & Pan et al. (2005): Data tracking dopamine responses during the learning process.
  • Schultz et al. (1993): The classic studies that originally linked dopamine to RPE.

Key Insights and Ideas

  • Time is Learned, Not Given: In FLEX, the "internal clock" for a specific cue only develops if that cue is actually paired with a reward. This solves the scalability problem of TD learning.
  • Integrated Dopamine Can Increase: A major "aha" moment is the discovery that the total amount of dopamine released during a trial can actually increase as an animal learns. Standard TD theory predicts that integrated dopamine should stay constant or decrease. FLEX matches the experimental reality where dopamine levels often surge as the temporal representation matures.
  • The "Moving Bump" is Optional: FLEX explains why the "moving bump" of activity (dopamine firing shifting from reward to cue) is seen in some experiments but not others. In FLEX, the cue response and the reward suppression evolve on different timescales based on circuit plasticity, rather than a literal step-by-step shift.
  • Retrospective Probability: FLEX suggests dopamine signaling might represent the retrospective probability that a reward was caused by a specific cue, aligning with recent "inference-based" views of the brain.

Key Results

  • Experimental Alignment: FLEX successfully replicates the "serial transfer of activation" (dopamine moving from reward to cue) seen in classic Pavlovian conditioning.
  • Quantitative Match: By reanalyzing data from Coddington and Dudman, the authors showed that the transient increase in integrated dopamine during learning—a phenomenon TD cannot explain—is a natural byproduct of the FLEX architecture.
  • Sequential Conditioning: The model handles complex "blocking" and "unblocking" tasks, where multiple cues are presented in sequence, matching the nuanced results of behavioral experiments that have long puzzled TD theorists.

Limitations and Future Work

  • Temporal Limits: The current implementation of FLEX relies on synaptic eligibility traces that last only a few seconds. Learning associations over minutes or hours would likely require different biological mechanisms (like protein synthesis or long-term memory systems) not yet in the model.
  • Anatomical Specificity: While the model assigns roles to the PFC and VTA, the exact "Timer" and "Messenger" cell types are theoretical constructs. Future experimental work using optogenetics or calcium imaging is needed to identify these specific populations in the brain.
  • Value Encoding: The current version of FLEX focuses on timing and prediction; the authors note that slight modifications would be needed to fully account for how the brain represents the magnitude or "value" of different rewards.

Summary updated

Abstract

The dominant theoretical framework to account for reinforcement learning in the brain is temporal difference learning (TD) learning, whereby certain units signal reward prediction errors (RPE). The TD algorithm has been traditionally mapped onto the dopaminergic system, as firing properties of dopamine neurons can resemble RPEs. However, certain predictions of TD learning are inconsistent with experimental results, and previous implementations of the algorithm have made unscalable assumptions regarding stimulus-specific fixed temporal bases. We propose an alternate framework to describe dopamine signaling in the brain, FLEX (Flexibly Learned Errors in Expected Reward). In FLEX, dopamine release is similar, but not identical to RPE, leading to predictions that contrast to those of TD. While FLEX itself is a general theoretical framework, we describe a specific, biophysically plausible implementation, the results of which are consistent with a preponderance of both existing and reanalyzed experimental data.
BibTeX
@inproceedings{cone2024learning,
  author = {Ian Cone and Claudia Clopath and Harel Z. Shouval},
  title = {{Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time}},
  booktitle = {Nature Communications},
  year = {2024},
  doi = {10.1038/s41467-024-50205-3},
}
Figure