Abstract
Understanding how the human brain processes naturalistic audiovisual information remains a central challenge in cognitive neuroscience. Progress has been limited, however, by the difficulty of modeling complex audiovisual feature spaces - most prior work has therefore relied on short, controlled stimuli, or on stimuli from one modality at a time, leaving cortical mechanisms that support real-world comprehension poorly characterized. Further, while recent advances in artificial intelligence now enable the extraction of high-dimensional, time-resolved features from naturalistic stimuli, how cortical regions dynamically process auditory and visual information as time unfolds remains largely unexplored. Using large-scale fMRI data collected while participants watched movies, we developed two complementary computational approaches relying on prediction performance to map the moment-by-moment dynamics of sensory processing across cortical regions: one tracks when one modality predicts a region's activity substantially better than the other, capturing temporal transitions in modality dominance, while the other identifies periods when both modalities predict the region well, indicating balanced representation of auditory and visual information. Together, these analyses reveal two complementary patterns of audiovisual organization across cortex: a pair of "bows" of modality switching - one posterior bow encircling category-selective visual cortex and another anterior bow spanning dorso-lateral frontal areas, and an arrow-like axis of bimodally predicted regions extending from lateral occipital cortex into the temporal cortex. The coexistence of these systems points to a cortical architecture that flexibly reweights sensory inputs while maintaining balanced multimodal representations, supporting robust comprehension of complex natural events. More broadly, this work illustrates how naturalistic neuroimaging experiments informed by modern machine learning approaches can reveal new principles of dynamic audiovisual processing in the human brain.
Introduction: The Challenge of Naturalistic Multimodality
In the real world, we don't experience "vision" or "hearing" in isolation; we navigate a seamless, multimodal stream of information. However, most cognitive neuroscience has historically relied on "impulse-response" style experiments—short, controlled beeps or flashes—or focused on a single modality at a time.
The challenge is that modeling the "feature space" of a movie is incredibly difficult. Quantifying the relationship between a character's dialogue, their facial expression, and the background music in a way that a computer can use to predict brain activity requires sophisticated tools. Because of this complexity, we haven't had a clear map of how the brain dynamically reweights these inputs as a story unfolds. This study bridges that gap by using modern AI to "read" movies and mapping those features to the brain's response.
Methods: Voxel-wise Encoding with AI Features
The researchers employed Voxel-wise Encoding Models, a technique where researchers attempt to predict fMRI BOLD responses based on features extracted from the stimuli.
AI Feature Extraction
To capture the nuance of naturalistic stimuli, the study used high-dimensional features from "Large Multimodal Models." Auditory features were extracted using Qwen2 Audio-7B, while visual features came from Qwen 2.5 VL and VideoMAE. These models capture everything from low-level acoustics to high-level semantic meaning and social intent.
Computational Paradigms
Two complementary approaches were developed to map sensory dynamics:
- Dominance Indication Paradigm (DIP): This identifies "switching" regions—areas where one modality (e.g., audio) predicts brain activity significantly better than the other at specific moments.
- Performance Indication Paradigm (PIP): This identifies "jointly predicted" regions where both modalities consistently provide a good fit, suggesting integrated processing.
Additionally, a custom Multimodal Transformer was built to predict brain activity by learning to assign time-varying "attention weights" to audio and video inputs for each brain region. Crucially, the model was given no anatomical priors; it had to "learn" which parts of the brain cared about which modality.
Key Insights: Two Systems for Audiovisual Processing
The most striking finding is the spatial organization of these mechanisms. The "switching" regions form two bow-shaped bands: one posterior (encircling the visual cortex) and one anterior (in the frontal lobes). Meanwhile, the "jointly predicted" regions form an arrow-like axis extending from the lateral occipital cortex into the temporal lobe.
Dynamic Reweighting vs. Static Integration
Traditionally, regions like the precuneus or angular gyrus were thought to be "amodal" (processing abstract concepts regardless of source). This paper suggests they are actually dynamic switchers that transiently prioritize whichever modality is most informative at that moment in the story. This switching doesn't happen millisecond-by-millisecond based on low-level flashes or bangs. Instead, it occurs over tens of seconds, suggesting it is driven by high-level narrative comprehension and "situation models."
AI as a "Zero-Shot" Neuroanatomist
The multimodal transformer, despite having no knowledge of brain anatomy, independently "discovered" the brain's canonical sensory structure. It automatically assigned high video weights to the occipital lobe and high audio weights to the temporal language areas, validating the model's biological relevance.
Key Results
- Switching Leaders: The study identified 13 "leader" regions for modality switching, including the precuneus, superior parietal lobule, and lateral prefrontal cortex.
- Joint Leaders: 8 regions were identified as leaders in joint prediction, forming a continuous axis that includes the Superior Temporal Sulcus (STS), a known hub for audiovisual integration.
- Variance Structure: Variance decomposition confirmed that switching regions rely heavily on the unique features of the dominant modality during their "on" periods. In contrast, joint regions maintain a balanced reliance on both modalities.
- Predictive Power: The encoding models achieved robust Pearson correlation ($r$) values, particularly in posterior visual and temporal areas, proving that AI-extracted features are highly effective at mapping naturalistic brain activity.
Limitations and Future Work
While the AI models are powerful, it is difficult to disentangle exactly which features (e.g., a specific word vs. the tone of voice) are driving the prediction. The internal representations of these LLMs often conflate low-level sensory data with high-level semantics. Furthermore, the use of the Harvard-Oxford atlas is relatively coarse; future work using higher-resolution, subject-specific parcellations could reveal even more intricate functional distinctions. Finally, while the inclusion of action-heavy movies helps balance the dialogue-heavy FRIENDS dataset, more diverse stimuli would further strengthen the findings.
Summary updated