The value of what’s to come: neural mechanisms coupling prediction error and reward anticipation

Kiyohito Iigaya, Tobias U. Hauser, Zeb Kurth‐Nelson, John P. O’Doherty, Peter Dayan, Raymond J. DolanbioRxiv 2019 · DOI

Abstract

Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can often be more gratifying than the very experience itself. Theories of anticipation have described how it induces behaviors ranging from beneficial information-seeking through to harmful addiction. However, it remains unclear how neural systems compute an attractive value from anticipation, instead of from the reward itself. To address this gap, we administered a decision-making task to human participants that allowed us to analyze brain activity during receipt of information predictive of future pleasant outcomes. Using a computational model of anticipatory value that captures participants’ decisions, we show that an anticipatory value signal is orchestrated by influences from three brain regions. Ventromedial prefrontal cortex (vmPFC) tracks the value of anticipation; dopaminergic midbrain responds to information that enhances anticipation, while sustained hippocampal activity provides a functional coupling between these regions. This coordinating function of the hippocampus is consistent with its known role in episodic future thinking. Our findings shed new light on the neural underpinnings of anticipation’s influence over decision-making, while also unifying a range of phenomena associated with risk and time-delay preference.

Introduction

Standard economic theory suggests that a reward's value should decrease the longer we have to wait for it—a concept known as temporal discounting. However, human behavior often contradicts this. We frequently postpone pleasant experiences (like saving a special dessert for tomorrow) or pay for information about a future reward even when that information doesn't change the outcome.

The gap in our understanding lies in the "utility of anticipation." While we know the brain computes the value of the reward itself, we haven't clearly mapped how it computes the value of waiting for that reward. Why is the "looking forward" part sometimes better than the "having" part? The authors sought to uncover the neural mechanisms that turn a delay from a cost into a source of pleasure.

Methods

The researchers combined a behavioral task, computational modeling, and functional magnetic resonance imaging (fMRI) in a study of 39 self-declared heterosexual male participants recruited from the University College London (UCL) community.

  1. The Task: Participants chose between "Immediate Information" (knowing right away if a reward was coming after a delay) or "No Information" (staying in the dark until the end). The rewards were appetitive images of attractive female models, and the delays ranged from 1 to 40 seconds.
  2. Computational Modeling: The team used a "utility of anticipation" model. Crucially, they introduced a "boosting" mechanism: when a participant receives a reward-predictive cue, it generates an anticipation-reward prediction error (aRPE). This "surprise" doesn't just signal a change in expectation; it actively boosts the value of the entire waiting period.
  3. fMRI Analysis: Model-predicted value signals were used to identify matching patterns in brain activity. Psychophysical Interaction (PPI) analysis was employed to examine functional connectivity between brain regions during the anticipation phase.
Task design and computational model of anticipatory utility.

Key Results

Behavioral Preference for Information

Participants showed a significantly stronger preference for advance information as the delay increased. They wanted to know their fate early so they could spend more time "savoring" the upcoming reward, confirming that information has intrinsic value when it enables anticipation.

The Neural "Savoring" Circuit

The study identifies a specialized three-region network responsible for computing anticipatory value:

  • vmPFC (Ventromedial Prefrontal Cortex): Acts as the "common currency" tracker, representing the actual value of the anticipation. The vmPFC BOLD signal significantly correlated with the model’s predicted anticipatory value ($p < 0.05$, whole-brain corrected).
  • Dopaminergic Midbrain (VTA/SN): Signals the "surprise" (aRPE) when information is received, acting as the trigger to boost the value of the wait.
  • Hippocampus: Acts as the "bridge." It provides sustained activity that couples the midbrain's surprise with the vmPFC's value signal.
Neural correlates of anticipatory value in the vmPFC.

Imagination and Individual Differences

The involvement of the hippocampus—a region central to memory and "episodic future thinking"—suggests that the value of anticipation is fueled by the ability to vividly imagine the coming reward. Furthermore, the strength of the functional coupling between the hippocampus, midbrain, and vmPFC predicted an individual's "boosting coefficient," or how much they personally valued anticipation in their choices.

Discussion

These findings suggest that anticipation is an active computation rather than a passive wait. The brain calculates a dynamic value signal that ramps up as a reward approaches, subject to both discounting and boosting from predictive information.

While the study's demographic was specific to the reward type used, the identified circuit provides a robust framework for future research. Understanding the "utility of anticipation" has significant implications for clinical work; for instance, "over-boosting" of anticipatory value could explain pathological behaviors in gambling and addiction. Future studies should investigate whether a similar circuit governs "dread" for aversive outcomes and how these mechanisms generalize across different reward types and populations.

Summary updated

Abstract

Having something to look forward to is a keystone of well-being. Anticipation of a future reward, like an upcoming vacation, can often be more gratifying than the very experience itself. Theories of anticipation have described how it induces behaviors ranging from beneficial information-seeking through to harmful addiction. However, it remains unclear how neural systems compute an attractive value from anticipation, instead of from the reward itself. To address this gap, we administered a decision-making task to human participants that allowed us to analyze brain activity during receipt of information predictive of future pleasant outcomes. Using a computational model of anticipatory value that captures participants’ decisions, we show that an anticipatory value signal is orchestrated by influences from three brain regions. Ventromedial prefrontal cortex (vmPFC) tracks the value of anticipation; dopaminergic midbrain responds to information that enhances anticipation, while sustained hippocampal activity provides a functional coupling between these regions. This coordinating function of the hippocampus is consistent with its known role in episodic future thinking. Our findings shed new light on the neural underpinnings of anticipation’s influence over decision-making, while also unifying a range of phenomena associated with risk and time-delay preference.
BibTeX
@misc{iigaya2019value,
  author = {Kiyohito Iigaya and Tobias U. Hauser and Zeb Kurth‐Nelson and John P. O’Doherty and Peter Dayan and Raymond J. Dolan},
  title = {{The value of what’s to come: neural mechanisms coupling prediction error and reward anticipation}},
  year = {2019},
  doi = {10.1101/588699},
  howpublished = {bioRxiv},
}
Figure