The neural basis of event segmentation: Stable features in the environment are reflected by neural states

Djamari Oetringer, Dora Gözükara, Umut Güçlü, Linda Geerligs

Abstract Our senses receive a continuous stream of complex information. Parsing this information into meaningful events allows us to extract relevant information, remember it, and act upon it. Previous research has related these events to so-called neural states: temporally and regionally specific stable patterns of brain activity, which tend to coincide with events in the stimulus. Neural states show a temporal cortical hierarchy: short states are present in early sensory areas, while longer states can be found in higher-level areas. Here we investigated what these neural states represent. We hypothesized that states at different levels of the cortical hierarchy are shaped by aspects of the stimulus to which these brain areas are responsive. To test this hypothesis, we analyzed fMRI data of participants watching a movie, using a data-driven method to identify the neural states. We found support for the aforementioned hypothesis: specifically the parahippocampal place area and retrosplenial cortex, known to be sensitive to places, showed an alignment between neural state boundaries and moments in the movie with a change in location, independent of changes in visual features and other covariates. These findings suggest that neural states reflect stable features in the (internal model of) the external environment, and that the cortical temporal hierarchy partly reflects the temporal scales at which representations of the environment evolve.

Introduction: Segmenting the Continuous Flow of Experience

Our brains are constantly bombarded by a "firehose" of sensory information. To navigate the world effectively, we must perform event segmentation—the process of chunking this continuous input into discrete, meaningful units. Neuroscientists have linked this process to "neural states," which are periods where brain activity patterns remain stable before shifting at an "event boundary."

While it is established that these states follow a temporal hierarchy—with early sensory areas exhibiting rapid shifts and higher-level areas maintaining stability for longer durations—a fundamental question remains: What do these neural states actually represent? This study investigates whether neural states are generic responses to change or if they are specifically "tuned" to the features a brain region is specialized to process.

Methodology

Researchers utilized the StudyForrest dataset, analyzing fMRI data from 15 participants watching the movie Forrest Gump. The analysis employed several key techniques:

  1. Greedy State Boundary Search (GSBS): A data-driven algorithm used to identify neural states by detecting significant shifts in brain activity patterns.
  2. Region of Interest (ROI) Analysis: The study focused on the Parahippocampal Place Area (PPA) and Retrosplenial Cortex (RSC)—regions specialized for spatial processing—and Early Visual (EV) areas for low-level feature comparison.
  3. Partial Correlation: To isolate the effect of location changes, the authors controlled for visual complexity (via AlexNet), low-level visual features (perceptual hashes), camera shots, and subjective event boundaries.
  4. Spatial Scales: Location changes were analyzed at two scales: "locales" (small-scale, e.g., a kitchen) and "settings" (large-scale, e.g., a house).

Key Results

The findings suggest that a region's segmentation "clock" is set by the specific features it processes. The right PPA showed a highly significant alignment between neural state boundaries and location changes ($p < 0.001$ for small-scale locations), even after controlling for all visual covariates. In contrast, Early Visual (EV) areas were indifferent to location changes but aligned significantly with low-level visual shifts.

Alignment of neural state boundaries with location changes across different brain regions and spatial scales.

Beyond spatial features, higher-level areas like the angular gyrus and medial prefrontal cortex aligned with subjective event boundaries—the "gist" of the story—rather than physical scene changes. Interestingly, camera cuts (shots) did not uniquely drive segmentation once visual features were controlled for, suggesting the brain tracks underlying environmental features rather than cinematic editing.

Discussion: The Origin of the Temporal Hierarchy

These results suggest that the brain's observed temporal hierarchy (short states in sensory areas, long states in higher areas) mirrors the natural statistics of the environment. Low-level features like edges and colors change rapidly as we move our eyes, while high-level concepts like location and narrative context evolve much more slowly. Neural states, therefore, likely represent the brain's "internal model," which remains stable as long as the current context is relevant for action and memory.

Limitations and Future Directions

The study's reliance on a small sample size ($N=15$) necessitated group-averaging, which may obscure individual differences in segmentation. Furthermore, while the focus was on visual and spatial features, the auditory cortex also showed unexplained segmentation, suggesting a need for future research into how auditory features (e.g., dialogue, music) contribute to neural state shifts. Finally, while the alignment is robust, future studies using high-temporal resolution techniques like MEG or ECoG could better map the precise timing of these transitions.

Summary updated

Abstract

Our senses receive a continuous stream of complex information. Parsing this information into meaningful events allows us to extract relevant information, remember it, and act upon it. Previous research has related these events to so-called neural states: temporally and regionally specific stable patterns of brain activity, which tend to coincide with events in the stimulus. Neural states show a temporal cortical hierarchy: short states are present in early sensory areas, while longer states can be found in higher-level areas. Here we investigated what these neural states represent. We hypothesized that states at different levels of the cortical hierarchy are shaped by aspects of the stimulus to which these brain areas are responsive. To test this hypothesis, we analyzed fMRI data of participants watching a movie, using a data-driven method to identify the neural states. We found support for the aforementioned hypothesis: specifically the parahippocampal place area and retrosplenial cortex, known to be sensitive to places, showed an alignment between neural state boundaries and moments in the movie with a change in location, independent of changes in visual features and other covariates. These findings suggest that neural states reflect stable features in the (internal model of) the external environment, and that the cortical temporal hierarchy partly reflects the temporal scales at which representations of the environment evolve.
BibTeX
@inproceedings{oetringer2024neural,
  author = {Djamari Oetringer and Dora Gözükara and Umut Güçlü and Linda Geerligs},
  title = {{The neural basis of event segmentation: Stable features in the environment are reflected by neural states}},
  booktitle = {Imaging},
  year = {2024},
  doi = {10.1162/imag\_a\_00432},
}
Figure