Anticipation of temporally structured events in the brain

Caroline S Lee, Mariam Aly, Christopher BaldassanoeLife 2021 · DOI

TL;DR

This study reveals that the human brain anticipates future events in a structured hierarchy, with posterior brain regions predicting the immediate future and more anterior regions predicting further ahead. By analyzing fMRI data from participants watching a movie repeatedly, researchers found that neural activity patterns shifted earlier with learning, demonstrating a predictive coding mechanism that aligns with subjective event boundaries.

What Problem Does This Paper Address?

Our ability to anticipate future events is crucial for adaptive behavior, allowing us to plan and react effectively. While previous research has explored how the brain predicts simple, discrete stimuli, it's less clear how this predictive capability extends to complex, naturalistic environments with rich temporal structures. Specifically, this paper investigates whether the brain exhibits a hierarchical organization for anticipating upcoming events, similar to how it processes past information, and how this anticipation relates to our subjective experience of event boundaries.

Methods

The researchers employed functional magnetic resonance imaging (fMRI) to capture brain activity while participants watched a 90-second movie clip six times. To analyze the temporal dynamics of brain activity, they utilized a searchlight approach combined with a Hidden Markov Model (HMM). This HMM was applied jointly across all viewings to identify a shared sequence of event patterns. By comparing the timing of these patterns between the first and subsequent viewings, they could quantify the degree of anticipation – how much earlier neural activity shifted to predict upcoming events. They also compared the brain's identified event boundaries with those manually annotated by human observers to link neural anticipation to subjective experience.

Datasets

The study used a publicly available dataset from the movie "The Grand Budapest Hotel," collected by Aly et al. (2018). This dataset included fMRI data from 30 participants watching a 90-second clip of the movie six times. Additionally, a separate group of 14 participants provided human annotations of event boundaries within the same movie clip.

Key Insights and Ideas

  • Hierarchical Anticipation: The brain doesn't just predict the immediate future; it does so across different timescales, organized along a posterior-to-anterior axis. This suggests a sophisticated predictive system that spans different levels of temporal abstraction.
  • Learning-Driven Prediction: Repeated exposure to a structured environment (the movie) leads to anticipatory shifts in neural activity. This highlights the brain's remarkable ability to learn and adapt its predictive models based on experience.
  • Bridging Neural and Subjective Experience: The study successfully linked the brain's internal event segmentation with human-labeled event boundaries. This connection is crucial for understanding how our subjective perception of events is grounded in neural processing.
  • Dynamic Event Boundaries: The timing of the brain's event boundaries shifts with learning, coming to precede human-annotated boundaries. This suggests that the brain actively "gets ahead" of perceived events, rather than just reacting to them.

Key Results

  • Anticipation Timescales: Neural activity patterns in many cortical regions shifted to precede initial viewing by up to 15 seconds with repeated viewings.
  • Posterior-to-Anterior Hierarchy: Anticipation varied hierarchically, with posterior regions showing less anticipation (predicting ~1-4 seconds ahead) and anterior regions showing more anticipation (predicting up to ~15 seconds ahead). This hierarchy was robust even when considering unthresholded statistical maps.
  • Event Boundary Alignment: With repeated viewing, the brain's event boundaries began to precede human-annotated boundaries by an average of 1-4 seconds. This shift was observed in specific regions like the middle temporal gyrus, fusiform gyrus, and superior temporal sulcus.

Limitations and Future Work

The authors acknowledge that their study relied on a single movie clip, and future work should explore how anticipation scales with movie duration and playback speed. They also suggest that the content of the movie and the frequency of event boundaries might influence anticipation amounts, and that important differences across stimuli may exist depending on salient features. Future research could also investigate shorter timescales using other neuroimaging modalities, longer timescales with more extensive narratives, and the impact of top-down goals on anticipation hierarchies using different tasks or brain stimulation studies. The study also opens avenues for integrating their findings with theories like the "successor representation" model to better understand how predictive representations are formed.

Summary updated

Abstract

Learning about temporal structure is adaptive because it enables the generation of expectations. We examined how the brain uses experience in structured environments to anticipate upcoming events. During fMRI (functional magnetic resonance imaging), individuals watched a 90 s movie clip six times. Using a hidden Markov model applied to searchlights across the whole brain, we identified temporal shifts between activity patterns evoked by the first vs. repeated viewings of the movie clip. In many regions throughout the cortex, neural activity patterns for repeated viewings shifted to precede those of initial viewing by up to 15 s. This anticipation varied hierarchically in a posterior (less anticipation) to anterior (more anticipation) fashion. We also identified specific regions in which the timing of the brain's event boundaries was related to those of human-labeled event boundaries, with the timing of this relationship shifting on repeated viewings. With repeated viewing, the brain's event boundaries came to precede human-annotated boundaries by 1-4 s on average. Together, these results demonstrate a hierarchy of anticipatory signals in the human brain and link them to subjective experiences of events.
BibTeX
@inproceedings{lee2021anticipation,
  author = {Caroline S Lee and Mariam Aly and Christopher Baldassano},
  title = {{Anticipation of temporally structured events in the brain}},
  booktitle = {eLife},
  year = {2021},
  doi = {10.7554/elife.64972},
}
Figure