Reward-Bases: Dopaminergic Mechanisms for Adaptive Acquisition of Multiple Reward Types

Beren Millidge, Yuhang Song, Armin Lak, Mark E. Walton, Rafał BogaczbioRxiv 2023 · DOI

TL;DR

This paper proposes a novel "Reward Bases" model that extends standard reinforcement learning to explain how animals can flexibly adapt their behavior to changing needs (like hunger or thirst) by learning separate value functions for different reward types. This model predicts, and experimental data supports, that different dopamine neurons are selectively tuned to these distinct reward dimensions, offering a more nuanced view of dopamine's role in motivation and learning.

What Problem Does This Paper Address?

The dominant theory of dopamine function in reinforcement learning (RL) assumes that animals learn a single value for each state, which is updated based on reward prediction errors. However, this model struggles to explain a key observation: animals can instantly adapt their preferences for different rewards based on their current physiological state (e.g., becoming more attracted to water when thirsty) without needing direct experience of the reward in that new state. Standard RL models would require repeated exposure to update value estimates, failing to capture this rapid, flexible revaluation. This paper aims to bridge this gap by proposing a model that accounts for this behavioral flexibility and its underlying neural mechanisms, particularly the role of dopamine.

Methods

The core of the paper introduces the "Reward Bases" model, an extension of the standard Temporal Difference (TD) learning algorithm. Instead of learning a single value function for a state, this model learns multiple "value bases," each corresponding to a specific reward dimension (e.g., food, water). The overall value of a state is then a weighted sum of these value bases, where the weights are determined by the animal's current motivational state (e.g., hunger level).

Key aspects of the method include:

  • Decomposition of Reward: The total reward is treated as a linear combination of individual reward bases.
  • Parallel Value Learning: Each reward basis is learned independently using a standard TD learning rule, creating parallel value functions.
  • State-Dependent Weighting: Motivational drives (mi) dynamically modulate the contribution of each value basis to the overall value function, allowing for flexible adaptation to current needs.
  • Prediction Error Modulation: The model also explores a "state-modulated prediction error" version, where prediction errors are scaled by motivational state, further enhancing neural plausibility.

The authors also re-analyzed existing experimental data from primate dopamine neurons and simulated behavioral experiments to validate their model.

Datasets

The paper utilizes and re-analyzes existing datasets:

  • Lak et al. (2014): Data on primate dopamine neuron activity in response to stimuli predicting different volumes of juice or amounts of banana. This dataset was used to test the prediction that dopamine neurons are selectively tuned to different reward dimensions.
  • Papageorgiou et al. (2016): Data on dopaminergic responses in rats during reward devaluation tasks. This was used to demonstrate the model's ability to capture instant revaluation of dopamine signals based on physiological state.
  • Robinson and Berridge (2013): Behavioral data from rats showing instant reward revaluation in response to salt deprivation. This was used to demonstrate the model's ability to reproduce flexible behavioral adaptation.

The paper does not introduce new datasets but rather re-interprets existing ones through the lens of their proposed model.

Key Insights and Ideas

  • Dopamine as a Multi-Dimensional Signal: The model challenges the idea of dopamine signaling a single, monolithic reward prediction error. Instead, it suggests dopamine neurons can encode prediction errors for specific reward dimensions, allowing for more nuanced and flexible learning.
  • Instant Revaluation Explained: The Reward Bases model provides a parsimonious explanation for how animals can instantly shift preferences based on physiological state, without needing new reward experiences. This is achieved by dynamically weighting learned value bases.
  • Neural Plausibility: The model's architecture, with parallel processing units for different reward bases, maps well onto known basal ganglia circuitry, suggesting a biologically plausible mechanism.
  • Beyond Simple Value: The work highlights that dopamine's role might be more sophisticated than just signaling "good" or "bad," extending to representing the type and dimension of reward.

Key Results

  • Dopamine Neuron Selectivity: Re-analysis of Lak et al. (2014) data showed that a significant number of dopamine neurons exhibited differential selectivity for juice versus banana rewards, supporting the model's prediction.
  • Instant Generalization: The model successfully reproduced instant generalization to new physiological states in simulations, matching both dopaminergic responses (Papageorgiou et al., 2016) and behavioral changes (Robinson & Berridge, 2013).
  • Computational Efficiency: The Reward Bases model achieves performance comparable to successor representations (a model known for generalization) but with significantly less memory storage.
  • Model Comparison: The Reward Bases model outperformed standard TD learning and a "homeostatic model" in a room navigation task requiring adaptation to changing reward values.

Limitations and Future Work

  • Known Weights: The current model assumes the motivational weights ($m_i$) are known. Future work should explore how the brain learns or infers these weights, potentially linking them to homeostatic set-points or drive reduction theories.
  • Reward Dimension Identification: While the model demonstrates the principle, identifying the full set of relevant reward dimensions represented in the brain remains an open question.
  • Topographical Organization: The paper suggests that neurons encoding specific dimensions might be topographically organized, which warrants further investigation.
  • Learning Novel Dimensions: The ability of dopamine neurons to learn entirely new reward dimensions is also a direction for future research.
  • Mixed Selectivity: While the paper introduces a "mixed selectivity" model to account for neurons responding to a combination of reward dimensions, it's not fully explored in the main simulations due to data limitations. Further investigation into this mixed selectivity is suggested.

Summary updated

Abstract

Abstract Animals can adapt their preferences for different types for reward according to physiological state, such as hunger or thirst. To describe this ability, we propose a simple extension of temporal difference model that learns multiple values of each state according to different reward dimensions such as food or water. By weighting these learned values according to the current needs, behaviour may be flexibly adapted to present demands. Our model predicts that different dopamine neurons should be selective for different reward dimensions. We reanalysed data from primate dopamine neurons and observed that in addition to subjective value, dopamine neurons encode a gradient of reward dimensions; some neurons respond most to food rewards while the others respond more to fluids. Moreover, our model reproduces instant generalization to new physiological state seen in dopamine responses and in behaviour. Our results demonstrate how simple neural circuit can flexibly optimize behaviour according to animals’ needs.
BibTeX
@misc{millidge2023reward,
  author = {Beren Millidge and Yuhang Song and Armin Lak and Mark E. Walton and Rafał Bogacz},
  title = {{Reward-Bases: Dopaminergic Mechanisms for Adaptive Acquisition of Multiple Reward Types}},
  year = {2023},
  doi = {10.1101/2023.05.09.540067},
  howpublished = {bioRxiv},
}

Loading related content...

Figure