The modulation of savouring by prediction error and its effects on choice

Kiyohito Iigaya, Giles W. Story, Zeb Kurth‐Nelson, Raymond J. Dolan, Peter DayaneLife 2016 · DOI

TL;DR

Why do we often want to know our fate in advance, even when it doesn't change the outcome? This paper proposes that "reward prediction errors" (RPEs) – the difference between expected and actual rewards – boost the pleasurable anticipation of future rewards. They developed a reinforcement learning model that explains this phenomenon and predicts why we might engage in riskier behaviors, like gambling, when there's a delay to the reward.

What Problem Does This Paper Address?

Humans and animals often exhibit a preference for information, even when that information is costly and doesn't alter the actual outcome. This "observing" or "information-seeking" behavior seems irrational from a purely outcome-based perspective. While it's known that anticipating rewards can be pleasurable (savoring), existing models struggle to explain why this anticipation drives seemingly irrational choices, especially when the delay to the reward is a factor. The authors aim to bridge this gap by proposing a mechanism where prediction errors, rather than just the anticipation itself, play a crucial role in modulating this preference.

Methods

The core of this research is a novel reinforcement learning (RL) model that incorporates "RPE-boosted anticipation." Standard RL models focus on learning values based on prediction errors to optimize future rewards. This model extends that by positing that the magnitude of the prediction error associated with a predictive cue can directly influence the subjective value of the anticipation itself. Specifically, positive RPEs (better than expected) boost savoring, while negative RPEs (worse than expected) can boost dread.

They tested this model using:

  • Mathematical modeling: Developing a formal RL framework with RPE-boosted anticipation.
  • Analysis of existing data: Re-analyzing published behavioral and neural data from monkeys.
  • New behavioral experiments: Conducting two experiments with human participants to test specific predictions of their model, particularly the influence of delay on information-seeking preferences.

Datasets

The paper utilizes:

  • Existing datasets: Behavioral and neural data from macaque monkeys published in Bromberg-Martin and Hikosaka (2011).
  • New human behavioral data: Data collected from two experiments involving human volunteers. Experiment 1 involved 14 participants, and Experiment 2 involved 31 participants. These experiments were designed to specifically test the delay-dependent preference for information.

Key Insights and Ideas

  • Prediction Errors as "Savoring Boosters": The most striking idea is that RPEs don't just signal learning; they actively modulate the pleasure derived from anticipating a reward. A positive surprise (unexpectedly good news) makes the waiting period even more enjoyable.
  • Delay is Key: The preference for information isn't constant. It increases with the delay to the reward, suggesting that the anticipation phase becomes more valuable when it's longer, especially if it's punctuated by prediction errors.
  • Explaining "Irrational" Gambling: The model offers a compelling explanation for why people might choose a gamble with a lower expected value but more uncertainty (and thus potential for RPEs) over a sure thing, particularly when there's a delay.
  • Dread and Savoring Coexist: The model extends to negative outcomes, suggesting that prediction errors can also boost "dread." Human participants in the study assigned negative values to no-reward outcomes, indicating a complex interplay of savoring and dread.

Key Results

  • Monkey Data: The model successfully accounted for macaque monkeys' preference for more informative cues and the neural activity in the lateral habenula, which is known to signal prediction errors.
  • Human Data (Experiment 1 & 2): Human participants showed a significantly increased preference for informative cues as the delay to the reward increased. This preference was minimal at short delays but pronounced at longer delays (20-40 seconds), aligning with the model's predictions.
  • Model Comparison: The RPE-boosted anticipation model provided a significantly better fit to the human behavioral data than standard Q-learning models or anticipation models without RPE-boosting.
  • Negative Outcome Values: Human participants assigned negative values to no-reward outcomes, and the model successfully incorporated this "dread" component.

Limitations and Future Work

  • Generalizability of RPE-Boosting: While the model's behavior is robust across different functional forms of RPE dependence, the exact neural mechanisms of this boosting are not fully elucidated.
  • Adaptation to Delay: The study noted some evidence of adaptation to delay in a control experiment, but this was not the focus and warrants further investigation, especially in tasks with randomized delays.
  • Preference Reversal at Extreme Delays: The authors acknowledge they couldn't confirm their prediction of a preference reversal at extremely long delays (beyond 135 seconds) due to experimental constraints.
  • Neural Underpinnings: Further research is needed to explore the specific neural circuits and mechanisms underlying anticipation, savoring, and dread, and how RPEs modulate these processes. The authors point to areas like the basal forebrain, ventral striatum, and insula as potential candidates.
  • Gambling and Addiction: The authors suggest that their findings have direct implications for understanding gambling behavior and addiction, and future work could explore these links more deeply.

Summary updated

Abstract

When people anticipate uncertain future outcomes, they often prefer to know their fate in advance. Inspired by an idea in behavioral economics that the anticipation of rewards is itself attractive, we hypothesized that this preference of advance information arises because reward prediction errors carried by such information can boost the level of anticipation. We designed new empirical behavioral studies to test this proposal, and confirmed that subjects preferred advance reward information more strongly when they had to wait for rewards for a longer time. We formulated our proposal in a reinforcement-learning model, and we showed that our model could account for a wide range of existing neuronal and behavioral data, without appealing to ambiguous notions such as an explicit value for information. We suggest that such boosted anticipation significantly drives risk-seeking behaviors, most pertinently in gambling.
BibTeX
@inproceedings{iigaya2016modulation,
  author = {Kiyohito Iigaya and Giles W. Story and Zeb Kurth‐Nelson and Raymond J. Dolan and Peter Dayan},
  title = {{The modulation of savouring by prediction error and its effects on choice}},
  booktitle = {eLife},
  year = {2016},
  doi = {10.7554/elife.13747},
}
Figure