Abstract
Making decisions in sequentially structured tasks requires integrating distally acquired information. The extensive computational cost of such integration challenges planning methods that integrate online, at decision time. Furthermore, it remains unclear whether “offline” integration during replay supports planning, and if so which memories should be replayed. Inspired by machine learning, we propose that (a) offline replay of trajectories facilitates integrating representations that guide decisions, and (b) unsigned prediction errors (uncertainty) trigger such integrative replay. We designed a 2-step revaluation task for fMRI, whereby participants needed to integrate changes in rewards with past knowledge to optimally replan decisions. As predicted, we found that (a) multi-voxel pattern evidence for off-task replay predicts subsequent replanning; (b) neural sensitivity to uncertainty predicts subsequent replay and replanning; (c) off-task hippocampus and anterior cingulate activity increase when revaluation is required. These findings elucidate how the brain leverages offline mechanisms in planning and goal-directed behavior under uncertainty.
Introduction
Planning is a computationally demanding process. To make optimal decisions in complex environments, an agent must often simulate multiple future steps and integrate information acquired at different times and locations. Traditional cognitive theories have largely focused on "online" planning—the idea that this integration happens at the moment a choice is required. However, real-time computation of this sort is slow and cognitively taxing.
In the field of Artificial Intelligence, the Dyna architecture offers an alternative: agents use "offline" periods to replay past experiences, updating their internal models and pre-computing optimal strategies. While rodent studies have long documented "hippocampal replay" during sleep and quiet wakefulness, it has remained unclear whether humans utilize similar offline mechanisms to support goal-directed planning. This study investigates whether the human brain uses rest periods to "revalue" past actions based on new information and how it prioritizes which memories to replay.
Methods
The researchers recruited 24 human participants to perform a novel 2-step reward revaluation task while undergoing fMRI scanning. The task was structured into four distinct phases:
- Learning Phase: Participants learned a mental "map" where a starting state (Stage I) led to two different rooms (Stage II), each associated with specific rewards.
- Relearning Phase: Participants were informed that the rewards in the Stage II rooms had changed. Crucially, the starting Stage I state was never shown during this phase, preventing direct online association between the start and the new rewards.
- Rest Periods: 30-second "offline" breaks were interleaved throughout the Relearning phase to allow for potential memory replay.
- Test Phase: Participants were returned to the Stage I starting state and asked to make a choice. Optimal performance required integrating the newly learned Stage II rewards with the original knowledge of the task's transition structure.
To detect replay, the researchers employed Multivariate Pattern Analysis (MVPA). By using distinct visual categories (faces vs. scenes) for Stage I and Stage II stimuli, they trained a classifier to identify neural signatures of the Stage I stimulus. This allowed them to "read" whether participants were replaying the starting state during rest periods, even when it was not visually present.
Key Insights and Results
Prioritized Replay and Uncertainty
The study found that the brain does not replay memories randomly. Instead, it follows a principle similar to "Prioritized Sweeping" in machine learning. Unsigned prediction errors—essentially the "surprise" or uncertainty felt when a reward changes unexpectedly—act as a neural tag. When a reward update occurred, the brain tagged the related distal memories (Stage I) for replay during the subsequent rest period.
Replay Predicts Planning Success
A critical finding was the direct relationship between neural replay and behavioral performance. The magnitude of Stage I replay detected during rest significantly predicted how effectively a participant would "replan" their behavior in the final test phase ($rho = 0.54, p = 0.0068$). Participants who replayed the starting state while learning about new rewards were far more likely to make the optimal choice later.
The "Dyna" Brain and Neural Hubs
The results suggest the human brain implements a biological version of RL algorithms like Dyna, using offline simulation to offload the computational costs of planning. Several key brain regions were implicated in this process:
- Anterior Cingulate Cortex (ACC) and Basal Ganglia: These regions tracked surprise (prediction errors) and coordinated the prioritization of replay.
- Hippocampus: This region showed increased univariate activity specifically during rest periods when the task required revaluation, compared to control periods where rewards remained stable.
Discussion and Limitations
This research provides the first neural evidence in humans that offline rest periods are used to replay memories to update planning strategies. However, the study has limitations. The temporal resolution of fMRI is relatively slow, meaning the researchers could identify what was being replayed (the category) but not the specific sub-second sequences (e.g., State A followed by State B) often seen in rodent electrophysiology.
Future work should explore more complex decision trees to determine if the brain performs multi-step "reverse replay" and whether these mechanisms support the formation of abstract "schemas" or successor representations for faster generalization in dynamic environments.
Summary updated