Simultaneous representation of a spectrum of dynamically changing value estimates during decision making

David Meder, Nils Kolling, Lennart Verhagen, Marco K. Wittmann, Jacqueline Scholl, Kristoffer H. Madsen, Oliver J. Hulme, Timothy E.j. Behrens, Matthew F. S. Rushworth

TL;DR

This paper reveals that key brain regions, particularly the dorsal anterior cingulate cortex (dACC), don't just hold a single value estimate for decision-making, but rather a spectrum of estimates based on different timescales of past experience. These representations are not randomly distributed but show topographic organization and dynamically adapt to environmental volatility, suggesting a sophisticated mechanism for flexible decision-making.

What Problem Does This Paper Address?

How do we make decisions when the environment is constantly changing? We rely on past experiences to estimate the value of different choices, but the crucial question is which past experiences are most relevant. In a stable environment, long-term experience might be best. However, in a volatile environment, relying on outdated information could lead to poor choices. Previous research focused on how value estimates are updated after an outcome (value learning), often linking environmental volatility to a higher "learning rate" (LR). This study shifts the focus to how the brain represents these value estimates at the moment of decision-making, specifically investigating whether the brain maintains multiple value estimates based on different timescales of experience, and how these representations adapt to environmental changes.

Methods

The researchers employed a novel fMRI analysis approach. Instead of fitting a single computational model to behavior and then looking for its neural correlates, they independently fitted 30 different learning rates (LRs) to the fMRI data for each voxel. This allowed them to determine the "best-fitting LR" for each voxel, essentially identifying which timescale of past experience best explained the neural activity in that specific brain area. They then analyzed the spatial distribution of these best-fitting LRs, their dynamic changes between stable and volatile environments, and their inter-regional connectivity. A key innovation was the use of a General Linear Model (GLM) with 30 separate regressors, each representing a value estimate calculated with a different LR, allowing for a fine-grained analysis of temporal scaling in neural representations.

Datasets

The study used fMRI data from 17 human subjects performing a probabilistic reversal learning task. This task involved choosing between two stimuli with known reward magnitudes but unknown and changing reward probabilities. The experiment was divided into two sub-sessions: one with stable reward probabilities and another with volatile probabilities, with the order counterbalanced across subjects. No new benchmark datasets were introduced.

Key Insights and Ideas

  • A Spectrum of Value Estimates: The brain doesn't rely on a single, unified value estimate. Instead, regions like the dACC and IPL maintain a "spectrum" of value estimates, each reflecting a different timescale of past experience (i.e., different learning rates).
  • Topographic Organization: These diverse value representations aren't randomly scattered. They exhibit a spatial organization within brain regions, with some areas showing gradients of LRs along specific anatomical axes (e.g., rostrocaudal in dACC).
  • Dynamic Adaptation to Volatility: The brain actively adjusts these value representations based on environmental volatility. In volatile environments, there's a shift towards higher LRs (emphasizing recent experience), and in stable environments, lower LRs (emphasizing longer-term experience) become more prominent.
  • Combined Adaptation Mechanisms: In the dACC, adaptation to volatility involves both a shift in the dominant LR and an increase in the "signal strength" (beta-weight) of the relevant LR's representation. This suggests a robust mechanism for prioritizing the most adaptive value estimates.
  • LRs as an Organizing Principle for Connectivity: The temporal scaling of value estimates (represented by LRs) isn't just an internal organizational feature of individual brain regions; it also dictates how these regions interact. Voxels with similar best-fitting LRs tend to show stronger functional connectivity.

Key Results

  • The dorsal anterior cingulate cortex (dACC) and inferior parietal lobule (IPL) showed strong evidence for representing value estimates based on a spectrum of LRs.
  • These LRs were not randomly distributed but showed topographic organization within the dACC, with a gradient along the rostrocaudal axis.
  • In volatile environments, the best-fitting LRs in dACC and IPL voxels were significantly higher than in stable environments, indicating a dynamic shift towards prioritizing recent experience.
  • The dACC exhibited a combined adaptation mechanism: not only did the best-fitting LR shift with volatility, but the signal strength of the relevant LR also increased. The IPL showed a similar LR shift but less pronounced signal strength changes.
  • Interactions between dACC and IPL were organized by LR, with voxels representing similar timescales showing stronger connectivity.
  • Ventromedial prefrontal cortex (vmPFC), another region involved in value processing, showed weaker evidence for LR-based representations and less dynamic adaptation to volatility.

Limitations and Future Work

  • Trial-by-Trial Dynamics: The current analysis averages data over trials. Future work could explore the trial-by-trial nature of these representations to gain a deeper understanding of their dynamic behavior.
  • Integration Mechanism: While the study shows that multiple value estimates are computed in parallel, the precise mechanism by which these are integrated into a single decision remains to be fully elucidated.
  • vmPFC Limitations: The vmPFC showed weaker evidence for LR-based representations and less dynamic adaptation. Further research could explore why this region might process value information differently.
  • Beyond fMRI: While fMRI provides valuable insights into brain regions, future studies using single-unit recordings could offer a more direct view of how individual neurons encode these different timescales of value.

Summary updated

Abstract

Decisions are based on value expectations derived from experience. We show that dorsal anterior cingulate cortex and three other brain regions hold multiple representations of choice value based on different timescales of experience organized in terms of systematic gradients across the cortex. Some parts of each area represent value estimates based on recent reward experience while others represent value estimates based on experience over the longer term. The value estimates within these areas interact with one another according to their temporal scaling. Some aspects of the representations change dynamically as the environment changes. The spectrum of value estimates may act as a flexible selection mechanism for combining experience-derived value information with other aspects of value to allow flexible and adaptive decisions in changing environments.
BibTeX
@inproceedings{meder2017simultaneous,
  author = {David Meder and Nils Kolling and Lennart Verhagen and Marco K. Wittmann and Jacqueline Scholl and Kristoffer H. Madsen and Oliver J. Hulme and Timothy E.J. Behrens and Matthew F. S. Rushworth},
  title = {{Simultaneous representation of a spectrum of dynamically changing value estimates during decision making}},
  booktitle = {Nature Communications},
  year = {2017},
  doi = {10.1038/s41467-017-02169-w},
}
Figure