Reward prediction error does not explain movement selectivity in DMS-projecting dopamine neurons

Rachel S. Lee, Marcelo G. Mattar, Nathan F. Parker, Ilana B. Witten, Nathaniel D. DaweLife 2019 · DOI

TL;DR

Dopamine neurons have traditionally been thought to signal "reward prediction errors" (RPEs), crucial for learning. However, recent findings show these neurons also respond to movement. This study investigates whether movement signals are just a form of RPE or a distinct signal. They conclude that dopamine neurons in the dorsomedial striatum (DMS) carry both movement and value signals, but these are separable, suggesting dopamine plays a direct role in movement beyond just reward learning.

What Problem Does This Paper Address?

For decades, the prevailing view has been that dopamine (DA) neurons primarily encode "reward prediction errors" (RPEs) – the difference between expected and actual rewards. This RPE signal is vital for reinforcement learning, helping organisms adapt their behavior to maximize rewards. However, this view has been challenged by recent studies revealing that DA neurons also exhibit activity correlated with movement. This creates a puzzle: are these movement-related signals simply a manifestation of RPEs tied to specific actions, or do DA neurons have a more direct role in motor control? This paper aims to resolve this dichotomy by investigating whether the observed movement selectivity in DMS-projecting DA neurons can be explained by an "action-specific RPE" or if it represents a distinct, movement-related signal.

Methods

The researchers used fiber photometry to record calcium signals (a proxy for neural activity) from DA neurons projecting to the dorsomedial striatum (DMS) in mice. These mice were trained on a probabilistic reversal learning task, where they had to choose between two levers, each offering a different probability of reward. The key to their approach was to disentangle movement-related signals from value-related signals.

They employed two main strategies:

  1. Behavioral Modeling: A trial-by-trial Q-learning model was used to estimate the animals' predicted values (Q-values) for each choice. This allowed them to quantify the expected reward for each action on a given trial.
  2. Signal Decomposition: They analyzed the DA signals in relation to:
  3. Contralateral vs. Ipsilateral Choice: This captures the movement direction relative to the recording hemisphere, as previously reported.
  4. Chosen vs. Unchosen Value: This reflects the RPE or value signal associated with the action the mouse actually took, compared to the alternative.
  5. Interaction Terms: They looked for interactions between movement direction and value to see if one explained the other.

They also analyzed signals time-locked to different behavioral events (nose poke, lever presentation, lever press) and examined both DA terminals and cell bodies to understand the origin of these signals.

Datasets

The study utilized data from 17 male mice performing a probabilistic reversal learning task. The behavioral data was analyzed using a Q-learning mixed-effects model. The neural data consisted of calcium imaging recordings from VTA/SN::DMS terminals (12 recording sites) and cell bodies (7 recording sites). No new datasets were introduced; rather, existing data was re-analyzed with a novel analytical framework.

Key Insights and Ideas

  • Movement and Value are Separable: The most significant finding is that DA signals in DMS-projecting neurons are modulated by both movement direction and expected value, but these two components are distinct and can be dissociated.
  • Chosen Value, Not Contralateral Value: While DA signals do reflect expected value, this value is tied to the chosen action, not the contralateral action. This rules out the hypothesis that movement selectivity is simply an RPE for contralateral actions.
  • Movement Direction Dominates Selectivity: The observed contralateral choice selectivity in DMS DA neurons is better explained by the direction of movement itself, rather than by the predicted value of that movement.
  • Reversal of Selectivity with Movement: Critically, when signals were time-locked to the lever press (when the mouse's movement direction reverses), the DA signal's selectivity also reversed, further supporting a link to movement direction rather than a static value signal.
  • Cell Bodies vs. Terminals: The findings were consistent between DA cell bodies and their terminals, suggesting that the movement-related coding is intrinsic to the DA neurons themselves, not solely a downstream effect in the striatum.

Key Results

  • DMS-projecting DA neurons showed elevated activity for contralateral choices compared to ipsilateral choices, consistent with previous findings.
  • These neurons also responded to the predicted value of the chosen action, showing higher activity when the previous trial's outcome was rewarding.
  • However, the value modulation was related to the chosen action's value, not the contralateral action's value.
  • When analyzing signals time-locked to the lever press, the contralateral selectivity reversed shortly after the press, coinciding with a reversal in the mouse's movement direction. This reversal was not observed in the value modulation.
  • Statistical modeling confirmed that movement direction was a significant predictor of DA signals, and this effect could not be fully explained by value signals.

Limitations and Future Work

  • Fiber Photometry Limitations: The use of fiber photometry measures bulk calcium signals, which averages activity across many neurons. It's possible that individual neurons within this population might have more specialized coding, or that distinct populations encode movement and value. Future work using single-unit recordings would be beneficial.
  • Behavioral Monitoring: While the study infers movement direction changes based on the task structure and signal reversals, direct head tracking would provide more precise behavioral data to confirm these inferences.
  • Causal Manipulations: The current study is correlational. Future research could employ causal manipulations (e.g., optogenetics) to directly test the role of these DA signals in movement execution and learning.
  • Extended Computational Models: The authors suggest that future computational models need to incorporate both movement direction and RPE signals to fully capture the complexity of DA neuron function. They also propose that this framework could help explain other observed DAergic phenomena, such as "ramps" in activity.

Summary updated

Abstract

Although midbrain dopamine (DA) neurons have been thought to primarily encode reward prediction error (RPE), recent studies have also found movement-related DAergic signals. For example, we recently reported that DA neurons in mice projecting to dorsomedial striatum are modulated by choices contralateral to the recording side. Here, we introduce, and ultimately reject, a candidate resolution for the puzzling RPE vs movement dichotomy, by showing how seemingly movement-related activity might be explained by an action-specific RPE. By considering both choice and RPE on a trial-by-trial basis, we find that DA signals are modulated by contralateral choice in a manner that is distinct from RPE, implying that choice encoding is better explained by movement direction. This fundamental separation between RPE and movement encoding may help shed light on the diversity of functions and dysfunctions of the DA system.
BibTeX
@inproceedings{lee2019reward,
  author = {Rachel S. Lee and Marcelo G. Mattar and Nathan F. Parker and Ilana B. Witten and Nathaniel D. Daw},
  title = {{Reward prediction error does not explain movement selectivity in DMS-projecting dopamine neurons}},
  booktitle = {eLife},
  year = {2019},
  doi = {10.7554/elife.42992},
}
Figure