Associative memory retrieval modulates upcoming perceptual decisions

Aaron M. Bornstein, Mariam Aly, Samuel F. Feng, Nicholas B. Turk‐Browne, Kenneth A. Norman, Jonathan D. CohenbioRxiv 2017 · DOI

TL;DR

This paper reveals that our decisions aren't just based on what we see now, but also on what our brains are actively recalling from memory before we even see the stimulus. By modeling this dynamic interplay, the researchers show that memory retrieval actively shapes our perceptual choices, suggesting a continuous inference process that integrates past experiences with current sensory input.

What Problem Does This Paper Address?

How do our expectations influence our decisions? While it's known that expectations can speed up and improve decision-making, the source and dynamic nature of these expectations have been less clear. Traditional models often treat expectations as static or set before a decision begins. This paper tackles the question of whether expectations are dynamically formed through ongoing memory retrieval, and how this memory-based inference interacts with incoming sensory information to guide perceptual choices. They aim to bridge the gap between laboratory studies of isolated information sources (like memory or vision) and real-world decision-making, where multiple sources are integrated.

Methods

The researchers developed a novel "cue-guided perceptual inference task." Participants were first exposed to cue-stimulus pairings to build associative memories. Then, in a test phase, a cue was presented, followed by a delay period, and finally a noisy visual stimulus (the "flickering probe"). Participants could respond at any time.

Key methodological components include:

  • Multi-stage Sequential Sampling Model (MSDDM): This computational model was central to their hypothesis. It posits a two-stage process: an initial stage where memory is sampled based on the cue, and a second stage where this memory-derived information is integrated with incoming sensory evidence. Crucially, the output of the first stage dynamically influences the parameters of the second stage.
  • Model Comparison: They rigorously compared their MSDDM against simpler models (single DDM, two unconnected DDMs) using Bayesian Information Criterion (BIC) to determine the best fit for behavioral data.
  • fMRI and MVPA: In a second experiment, they used functional Magnetic Resonance Imaging (fMRI) and Multivariate Pattern Analysis (MVPA) to measure neural activity. This allowed them to quantify "reinstatement index" – the degree to which neural patterns in face/scene-selective brain regions (FFA/PPA) reflected the cued memory – and relate it to behavioral responses.

Datasets

The study used custom-generated datasets from their experimental tasks. No pre-existing datasets were used for training or evaluation.

Key Insights and Ideas

  • Memory is not passive recall; it's active inference: The study strongly suggests that memory retrieval isn't just about retrieving stored information, but about actively using that information to infer what's likely to happen next, influencing ongoing perceptual processing.
  • Expectations are dynamic and trial-by-trial: Unlike static prior beliefs, expectations in this task were shaped by the cue and the time available for memory sampling on each specific trial.
  • Memory and sensory evidence are integrated continuously: The MSDDM model, which best explained the data, implies that memory and sensory information are not processed in separate, isolated steps but are woven together in a continuous inference process.
  • Neural reinstatement predicts behavior: fMRI data showed that the degree to which neural patterns reflected the cued memory (reinstatement index) before the visual stimulus appeared predicted response times, particularly when memory and sensory evidence were in agreement.
  • The brain "pre-loads" perceptual processing: The findings suggest that memory retrieval before sensory input effectively sets the stage for how incoming visual information will be processed and interpreted.

Key Results

  • Behavioral Replication: Participants' choices and response times were well-explained by the MSDDM, showing that both memory cues and sensory evidence influenced their decisions. Response times were bimodal, with distinct peaks corresponding to responses before and after the visual probe onset, supporting the two-stage model.
  • Model Superiority: The MSDDM significantly outperformed simpler models, indicating the necessity of a dynamic, two-stage sampling process where the first stage influences the second.
  • Neural Correlates: The reinstatement index (neural similarity to the cued memory) was higher when memory sampling was more difficult (less predictive cues) and when participants responded later. This index predicted faster responses when the cued memory matched the subsequent sensory evidence (valid cues).
  • Sensory Strength Matters: The influence of memory on speeding up responses was more pronounced when sensory evidence was weak (low coherence), suggesting that memory plays a larger role in disambiguating uncertain sensory input.

Limitations and Future Work

  • Algorithmic Level Ambiguity: The authors acknowledge that their study, like most evidence-sampling research, cannot definitively distinguish between accumulation models and "extreme selection" models at the algorithmic level. Future work with higher temporal resolution neuroimaging could help clarify this.
  • Serial vs. Parallel Integration: While their model is consistent with both serial and parallel integration of evidence, further studies are needed to disentangle these possibilities, potentially by using cues with both statistical and semantic relationships.
  • Confidence and Weighting: The study did not directly measure or manipulate confidence in memory or sensory evidence, nor how these confidence levels might influence the weighting of each information source. Future extensions could explore this.
  • Understanding the "Cost" of Sampling: The reason for early responses (before sensory input) remains an open question, possibly related to a "cost" of waiting to act. Further work is needed to understand the internal trade-offs involved in sampling.
  • Clinical Implications: The findings suggest that biases, often seen as static, are constructed dynamically and could be altered. This has potential implications for understanding and treating conditions characterized by altered expectations.

Summary updated

Abstract

Abstract Expectations can inform fast, accurate decisions. But what informs expectations? Here we test the hypothesis that expectations are set by dynamic inference from memory. Participants performed a cue-guided perceptual decision task with independently-varying memory and sensory evidence. Cues established expectations by reminding participants of past stimulus-stimulus pairings, which predicted the likely target in a subsequent noisy image stream. Participant’s responses used both memory and sensory information, in accordance to their relative reliability. Formal model comparison showed that the sensory inference was best explained when its parameters were set dynamically at each trial by evidence sampled from memory. Supporting this model, neural pattern analysis revealed that responses to the probe were modulated by the specific content and fidelity of memory reinstatement that occurred before the probe appeared. Together, these results suggest that perceptual decisions arise from the continuous sampling of memory and sensory evidence.
BibTeX
@misc{bornstein2017associative,
  author = {Aaron M. Bornstein and Mariam Aly and Samuel F. Feng and Nicholas B. Turk‐Browne and Kenneth A. Norman and Jonathan D. Cohen},
  title = {{Associative memory retrieval modulates upcoming perceptual decisions}},
  year = {2017},
  doi = {10.1101/186817},
  howpublished = {bioRxiv},
}

Loading related content...

Figure