Slow-rising and fast-falling dopaminergic dynamics jointly adjust negative prediction error in the ventral striatum

Yu Shikano, Sho Yagishita, Kenji F. Tanaka, Norio TakatabioRxiv 2021 · DOI

TL;DR

This study reveals that the magnitude of dopamine dips in the mouse ventral striatum directly reflects the degree of reward expectation, particularly when those expectations are unmet. This finding provides a crucial link between behavioral anticipation, dopamine signaling, and negative prediction errors, offering a more nuanced understanding of how the brain learns from disappointment.

What Problem Does This Paper Address?

While it's well-established that dopamine (DA) activity signals positive reward prediction errors (better-than-expected outcomes), the neural mechanisms underlying negative prediction errors (worse-than-expected outcomes) are less understood. Specifically, how the brain quantitatively represents the magnitude of disappointment or unmet expectations remains an open question. This paper aims to bridge this gap by investigating whether dopamine dips in the ventral striatum can serve as a quantitative signal for the degree of unmet reward expectations.

Methods

The researchers employed a multi-pronged approach:

  • Behavioral Task: A fixed-ratio (FR5) lever-press task was designed where mice had to press a lever five times to receive a food reward. This setup allowed for the manipulation of reward expectations by varying the proximity to task completion.
  • Fiber Photometry with GRABDA2m: A genetically encoded dopamine sensor (GRABDA2m) was virally expressed in the ventral striatum of mice. This allowed for real-time, in vivo monitoring of extracellular dopamine levels using fiber photometry. This technique offers high sensitivity and specificity for dopamine detection.
  • Computational Modeling: An actor-critic reinforcement learning (RL) model was developed to simulate mouse behavior and dopamine dynamics. This model incorporated temporal difference (TD) errors, which are thought to be reflected in dopamine signals, and implemented state transitions to account for potential misrecognition of lever presses by the mice.

Datasets

Not explicitly detailed as separate datasets. The study utilized data collected from experiments involving mice performing the operant conditioning task and from the computational model simulations.

Key Insights and Ideas

  • Dopamine Dips as a Quantitative Signal for Disappointment: The most significant insight is that the amplitude of the dopamine dip, not just its presence, directly correlates with the degree of unmet reward expectation. This suggests a graded response to disappointment.
  • Premature Magazine Entries (PMEs) as a Proxy for Expectation: The frequency of PMEs (checking the food magazine before completing the task) served as a behavioral indicator of rising reward expectations as mice approached task completion.
  • Tonic and Phasic DA Dynamics Cooperate: The study suggests that both the pre-dip DA level (tonic) and the dip-trough level (phasic) dynamically interact to sculpt the overall DA dip amplitude, providing a more comprehensive picture of DA signaling during negative prediction errors.
  • RL Model Replicates Key Findings: The computational model successfully reproduced the observed correlation between PME frequency and DA dip amplitude, lending strong support to the RL framework for understanding these neural signals.

Key Results

  • Larger reward expectations (indicated by higher PME frequency) were associated with a slower rise and a larger subsequent decrease (dip) in dopamine levels in the ventral striatum.
  • A monotonic relationship was demonstrated between the DA dip amplitude and the degree of behavioral reward expectations.
  • The computational model showed that TD errors increased with PME group numbers, mirroring the experimental findings of larger DA dips.
  • The model also suggested that mice might occasionally misrecognize the number of lever presses, contributing to the observed behavioral variability.

Limitations and Future Work

  • Caveat on Session-Level Comparisons: The authors note that across sessions, the DA dip amplitude did not always directly correspond to the absolute PME rate, suggesting that the relationship is more robust within a task session and on a trial-by-trial basis.
  • Spatial Pattern of GRABDA2m Expression: The exact spatial match between GRABDA2m expression and DA receptor distribution in the striatum is unknown, raising a question about whether all neurons expressing the sensor can effectively detect the DA levels. However, they cite recent work suggesting MSNs can detect small DA level differences.
  • Virus Expression Stability: The study acknowledges the need for stable GRABDA2m expression, which was maintained for 3-4 weeks post-injection.
  • Future Directions: The authors suggest further investigations into the neurophysiological mechanisms linking DA neuron firing and extracellular DA dynamics to DA dip variations. They also highlight the need to explore what mechanisms underlie long-term DA dip dynamics in response to spontaneous behavior in operant tasks.

Summary updated

Abstract

Abstract The greater the reward expectations are, the more different the brain’s physiological response will be. Although it is well-documented that better-than-expected outcomes are encoded quantitatively via midbrain dopaminergic (DA) activity, it has been less addressed experimentally whether worse-than-expected outcomes are expressed quantitatively as well. We show that larger reward expectations upon unexpected reward omissions are associated with the preceding slower rise and following larger decrease (DA dip) in the DA concentration at the ventral striatum of mice. We set up a lever press task on a fixed ratio (FR) schedule requiring five lever presses as an effort for a food reward (FR5). The mice occasionally checked the food magazine without a reward before completing the task. The percentage of this premature magazine entry (PME) increased as the number of lever presses approached five, showing rising expectations with increasing proximity to task completion, and hence greater reward expectations. Fiber photometry of extracellular DA dynamics in the ventral striatum using a fluorescent protein (genetically encoded GPCR-activation-based-DA sensor: GRAB DA2m ) revealed that the slow increase and fast decrease in DA levels around PMEs were correlated with the PME percentage, demonstrating a monotonic relationship between the DA dip amplitude and degree of expectations. Computational modeling of the lever press task implementing temporal difference errors and state transitions replicated the observed correlation between the PME frequency and DA dip amplitude in the FR5 task. Taken together, these findings indicate that the DA dip amplitude represents the degree of reward expectations monotonically, which may guide behavioral adjustment.
BibTeX
@misc{shikano2021slow,
  author = {Yu Shikano and Sho Yagishita and Kenji F. Tanaka and Norio Takata},
  title = {{Slow-rising and fast-falling dopaminergic dynamics jointly adjust negative prediction error in the ventral striatum}},
  year = {2021},
  doi = {10.1101/2021.07.23.453499},
  howpublished = {bioRxiv},
}

Loading related content...

Figure