Reminders of past choices bias decisions for reward in humans

Aaron M. Bornstein, Mel Win Khaw, Daphna Shohamy, Nathaniel D. DawNature Communications 2017 · DOI

TL;DR

This paper challenges the prevailing view of decision-making as relying on averaged past rewards. Instead, it provides evidence that humans make choices by sampling individual past experiences, and that these memories can be deliberately cued to influence current decisions. This "sampling" architecture offers a more flexible and nuanced explanation for human behavior, particularly in situations with sparse or complex reward histories.

What Problem Does This Paper Address?

For decades, reinforcement learning (RL) models have dominated our understanding of how humans and animals learn from experience to make decisions. A cornerstone of these models is the idea of "incremental learning," where the value of an action is updated based on a running average of past rewards. This approach works well in tasks with frequent, repeated choices among a limited set of options. However, it struggles to explain decision-making in more naturalistic scenarios where experiences are sparse, and a single, impactful past event might be more relevant than a long history of averaged outcomes. The authors address this by proposing and testing an alternative: a "sampling" model where decisions are guided by retrieving and evaluating individual memories of past choices, rather than relying on aggregated statistics.

Methods

The researchers employed a two-pronged approach:

  1. Computational Modeling and Behavioral Data Analysis (Experiment 1): They re-analyzed data from a previously published "restless bandit" task. In this task, participants repeatedly chose between four slot machines with slowly changing reward probabilities. The core of this experiment was to compare two computational models:
  2. Temporal Difference (TD) Learning Model: This represents the standard incremental learning approach, maintaining a running average of rewards for each option.
  3. Sampling Model: This novel model estimates the value of an option by stochastically sampling individual past outcomes associated with that option, with a bias towards more recent experiences. They specifically tested a version that sampled just one past experience per decision.

    They then compared how well each model fit the participants' choice sequences and, for a subset of participants, their fMRI data (specifically, neural signals related to reward prediction errors (RPEs) and chosen values (CVs)).

  4. Experimental Manipulation and Behavioral Data Analysis (Experiment 2): To establish a causal link between individual memories and choices, they designed a new "ticket bandit" task. This was a simpler two-armed bandit task where each choice yielded a "ticket" (a unique photograph) associated with either a $5 reward or a $5 penalty. Crucially, they interspersed "recognition probe" trials where participants were shown a ticket and asked if they recognized it. This was designed to incidentally remind participants of the specific trial on which they encountered that ticket, including the choice made and the outcome received. They hypothesized that this reminder would bias subsequent choices towards the outcome of the cued past trial. They then modeled these choices using a modified sampling model that included a parameter for the influence of these "evoked" memories.

Datasets

  • Experiment 1: Behavioral and fMRI data from 20 participants (14 with fMRI data, 6 behavioral-only) performing a four-armed restless bandit task. This data was originally collected and analyzed in Daw et al. (2006).
  • Experiment 2: Behavioral data from 21 participants (after exclusions) performing a two-armed ticket bandit task with interspersed memory probes. This dataset appears to be newly collected for this study.

Key Insights and Ideas

  • Beyond Averaging: The Power of Episodic Memory: The most striking insight is that human decision-making might not solely rely on abstract, averaged reward values. Instead, we seem to actively retrieve and utilize specific, individual past experiences. This "episodic sampling" offers a more flexible mechanism for adapting to novel situations or complex environments where averaged statistics might be misleading or insufficient.
  • Incidental Reminders Have Causal Power: The second experiment provides strong evidence for the causal role of individual memories. By simply presenting participants with a visual cue (a ticket photograph), the researchers could reliably bias their subsequent choices, demonstrating that specific past outcomes, even those from distant trials, can be reactivated and influence current decisions.
  • A Unified Architecture for Learning and Memory: The findings suggest a tighter integration between episodic memory systems (like the hippocampus) and value-based learning systems (like the striatum) than previously assumed. This challenges the strict dichotomy between "model-based" (flexible, memory-driven) and "model-free" (habitual, average-based) learning.
  • Explaining Complex Choice Patterns: The sampling model naturally accounts for phenomena that are difficult for simple averaging models to explain, such as the "wavy recency effect" and the influence of rare but impactful past experiences.

Key Results

  • Experiment 1: The sampling model provided a significantly better fit to participants' choices than the standard TD learning model (mean log Bayes factor of 8.8867). This superiority extended to decision-related neural signals (RPEs and CVs), which were also better explained by the sampling model.
  • Experiment 2: Choices following a recognition probe were significantly influenced by the evoked past trial, even when that trial occurred many trials prior (e.g., 39 trials back). This influence was comparable to the effect of a reward received just 2-3 trials prior, highlighting the power of the reminder. The sampling model, augmented to include the influence of evoked memories, also provided a superior fit to the behavioral data compared to a standard TD model (mean log Bayes factor of 6.9182).

Limitations and Future Work

  • Sampling Mechanism Specificity: While the study demonstrates the utility of sampling, the precise mechanism by which samples are selected from memory remains an open question. The current model assumes recency-based sampling, but other similarity-based criteria might be more relevant in environments with more defined structures. Future work could explore these alternative sampling functions.
  • Hybrid Models: The authors acknowledge that real-world decision-making likely involves a combination of sampling and averaging mechanisms. While they explored a hybrid model in Experiment 2, the results were equivocal. Further investigation into how these systems interact and potentially buffer through a common prediction error signal is warranted.
  • Generalizability: The tasks used were simplified bandit problems. Future research should explore whether this episodic sampling architecture extends to more complex, real-world decision-making scenarios with richer contextual information.
  • Neural Correlates of Sampling: While neural signals were better explained by the sampling model, directly identifying the neural mechanisms of episodic retrieval and its integration into value estimation remains a key area for future investigation.

Summary updated

Abstract

We provide evidence that decisions are made by consulting memories for individual past experiences, and that this process can be biased in favour of past choices using incidental reminders. First, in a standard rewarded choice task, we show that a model that estimates value at decision-time using individual samples of past outcomes fits choices and decision-related neural activity better than a canonical incremental learning model. In a second experiment, we bias this sampling process by incidentally reminding participants of individual past decisions. The next decision after a reminder shows a strong influence of the action taken and value received on the reminded trial. These results provide new empirical support for a decision architecture that relies on samples of individual past choice episodes rather than incrementally averaged rewards in evaluating options and has suggestive implications for the underlying cognitive and neural mechanisms.
BibTeX
@inproceedings{bornstein2017reminders,
  author = {Aaron M. Bornstein and Mel Win Khaw and Daphna Shohamy and Nathaniel D. Daw},
  title = {{Reminders of past choices bias decisions for reward in humans}},
  booktitle = {Nature Communications},
  year = {2017},
  doi = {10.1038/ncomms15958},
}

Loading related content...

Figure