A feature-specific prediction error model explains dopaminergic heterogeneity

Rachel S. Lee, Yotam Sagiv, Ben Engelhard, Ilana B. Witten, Nathaniel D. DawbioRxiv 2022 · DOI

TL;DR

Dopamine neurons are thought to signal reward prediction errors (RPEs), but recent findings show they're surprisingly diverse in what they respond to. This paper proposes a "feature-specific RPE" model where individual dopamine neurons encode RPEs for different aspects of the environment's state, rather than a single global RPE. This new model successfully explains observed dopamine neuron heterogeneity and offers a more biologically plausible account of reinforcement learning in complex environments.

What Problem Does This Paper Address?

The classic model of dopamine (DA) neuron function posits that they signal a "reward prediction error" (RPE) – a scalar value indicating how surprising a reward is. This has been a highly successful framework, linking neural activity to learning and behavior. However, recent experimental evidence has challenged a core assumption: that this RPE signal is uniform and globally broadcast. Studies have shown that DA neurons in areas like the ventral tegmental area (VTA) exhibit significant heterogeneity, responding to different task features (like position, cues, or movement) even while showing more consistent responses to reward itself. This discrepancy between the simple RPE theory and complex neural responses creates a puzzle: how can DA neurons be so diverse yet still play a crucial role in learning based on reward prediction? Previous attempts to explain this heterogeneity, often termed "outcome-specific PEs," proposed parallel circuits for different outcomes, but these models struggle to explain why DA neurons respond uniformly to reward while being heterogeneous in other aspects.

Methods

The authors introduce a novel "feature-specific RPE" model. Instead of a single RPE signal, this model posits that DA neurons within a circuit (like the VTA) each compute an RPE based on a subset of inputs representing different features of the animal's state. The overall RPE signal is then reconstructed by summing these feature-specific RPEs.

To test this, they employed a deep reinforcement learning (RL) agent trained on a virtual reality T-maze navigation and decision-making task. This agent's internal state representations (derived from convolutional and LSTM layers processing visual input) were used to compute feature-specific RPEs. These model-generated RPEs were then compared to population recordings from DA neurons in mice performing a similar task. They also extended their model to explain heterogeneity in action responses observed in SNc DA neurons.

Datasets

The study uses two primary sources of data:

  1. Simulated data: Generated by a deep reinforcement learning agent trained on a virtual reality T-maze task. This agent's internal state representations and computed feature-specific RPEs serve as the model's output.
  2. Neural recordings: Data from previously published experiments (Engelhard et al., 2019) involving VTA DA neurons in mice performing a similar virtual T-maze task. They also reanalyzed data related to SNc DA neurons from other studies.

Key Insights and Ideas

  • Heterogeneity arises from state features, not just outcomes: The core idea is that DA neuron diversity stems from them encoding RPEs for different components of the state (e.g., position, cue identity) rather than just different reward outcomes.
  • A population code for RPE: Individual DA neurons don't necessarily signal a global RPE, but rather contribute to a population code where different neurons represent RPEs related to different features of the state.
  • Reconciling classic and modern findings: The feature-specific RPE model bridges the gap between the established RPE theory and recent observations of DA neuron heterogeneity, showing how the former can give rise to the latter.
  • Anatomical plausibility: The model aligns better with the known topographic organization of cortico-striatal circuits, where inputs are not uniformly converged.
  • Reward uniformity explained: The model predicts that while neurons might respond heterogeneously to task variables, their responses to the outcome (reward) should be more uniform, matching empirical observations.

Key Results

  • The feature-specific RPE model successfully recapitulated key aspects of VTA DA neuron heterogeneity observed in neural recordings, including heterogeneous responses to task variables (view angle, position, cues) during navigation and decision periods, alongside more uniform responses to reward at the outcome period.
  • The model demonstrated that incidental, reward-irrelevant visual information from the environment could propagate into feature-specific RPEs, even if it averaged out in the overall scalar RPE.
  • The model's predictions for cue responses were consistent with neural data, showing that responses were modulated by the reward consequence of cues (confirmatory vs. disconfirmatory), not just their presence.
  • The framework was extended to explain heterogeneity in SNc DA neurons, particularly the trade-off between movement and reward selectivity, and specific press-timing preferences in sequential tasks.

Limitations and Future Work

  • Simplifications in the model: The authors acknowledge that their deep RL model has simplifications, such as not varying running speed or other kinematics, and not carrying information across trials. This means it might not perfectly capture all aspects of neural data, particularly temporally extended responses like positional ramping.
  • Distinguishing feature vs. outcome heterogeneity: A key prediction is that feature-specific heterogeneity should be nested within projection-defined DA populations, while outcome-specific heterogeneity should be found between them. Formally testing this prediction by comparing cue and outcome responses across different DA projection populations is highlighted as a crucial future empirical test.
  • Credit assignment mechanisms: The paper discusses different computational mechanisms for credit assignment (e.g., common error signal vs. "mixtures of experts") and suggests that future work could explore how the brain implements these, potentially through behavioral experiments (like blocking), projection tracing, and neurophysiological studies.
  • Action Prediction Error (APE) evidence: While the feature-specific APE model shows promise for explaining SNc DA neuron behavior, the authors note that definitive evidence for DA neurons encoding APEs analogous to RPEs is still developing.

Summary updated

Abstract

The hypothesis that midbrain dopamine (DA) neurons broadcast an error for the prediction of reward (reward prediction error, RPE) is among the great successes of computational neuroscience 1–3 . However, recent results contradict a core aspect of this theory: that the neurons uniformly convey a scalar, global signal. For instance, when animals are placed in a high-dimensional environment, DA neurons in the ventral tegmental area (VTA) display substantial heterogeneity in the features to which they respond, while also having more consistent RPE-like responses at the time of reward 4 . We argue that the previously predominant family of extensions to the RPE model, which replicate the classic model in multiple parallel circuits, are ill-suited to explaining these and other results concerning DA heterogeneity within the VTA. Instead, we introduce a complementary “feature-specific RPE” model positing that DA neurons within VTA report individual RPEs for different elements of a population vector code for an animal’s state (moment-to-moment situation). To investigate this claim, we train a deep reinforcement learning model on a navigation and decision-making task and compare the feature-specific RPE derived from the network to population recordings from DA neurons during the same task. The model recapitulates key aspects of VTA DA neuron heterogeneity. Further, we show how our framework can be extended to explain patterns of heterogeneity in action responses reported among SNc DA neurons 5 . Thus, our work provides a path to reconcile new observations of DA neuron heterogeneity with classic ideas about RPE coding, while also providing a new perspective on how the brain performs reinforcement learning in high dimensional environments.
BibTeX
@misc{lee2022feature,
  author = {Rachel S. Lee and Yotam Sagiv and Ben Engelhard and Ilana B. Witten and Nathaniel D. Daw},
  title = {{A feature-specific prediction error model explains dopaminergic heterogeneity}},
  year = {2022},
  doi = {10.1101/2022.02.28.482379},
  howpublished = {bioRxiv},
}

Loading related content...

Figure