Neural dSCA: demixing multimodal interaction among brain areas during naturalistic experiments

Yu Takagi, Laurence T. Hunt, Ryu Ohata, Hiroshi Imamizu, Jun-ichiro Hirayama

Abstract

Mapping neural communication during complex, real-world experiences is a major challenge in neuroscience. While naturalistic experiments like movie-watching offer rich data, overlapping signals from different sensory modalities (e.g., audio and visual) often obscure specific functional interactions. This paper introduces Neural demixed Shared Component Analysis (NdSCA), a framework that combines deep learning with classical statistics to disentangle and map feature-specific brain interactions.

The Challenge of Naturalistic Experiments

Modern neuroscience is shifting from "sterile" laboratory tasks—such as responding to a single flashing dot—toward "naturalistic" experiments, like watching a feature film. While movies provide a more ecologically valid view of brain function, they present a significant data analysis challenge.

In a naturalistic setting, the brain processes dialogue, music, faces, and movement simultaneously. Standard tools like Canonical Correlation Analysis (CCA) or Reduced-Rank Regression (RRR) can identify that two brain areas are communicating, but they cannot specify what information is being shared. The signals are inherently "mixed," and the high-dimensional, nonlinear nature of naturalistic data often causes traditional statistical tools like ANOVA to fail.

Conceptual overview comparing traditional and naturalistic experiments, and the shift from conventional to demixed interaction analysis.

Methodology: The NdSCA Framework

NdSCA addresses this by bridging the gap between deep learning and interpretable dimensionality reduction. The method operates in two primary stages:

  1. Neural Decomposition (ND): The framework utilizes a Conditional Variational Autoencoder (CVAE) to perform a "Neural ANOVA." This stage decomposes the activity of a target brain area into additive components: signals driven by specific features (e.g., audio or video) and a "latent" component representing unobserved or spontaneous brain activity.
  2. Cross-Areal Interaction: Once signals are demixed, the model employs Reduced-Rank Regression (RRR) to identify the low-dimensional "bottleneck" of communication between a source brain area and these specific demixed components in the target area.

For simpler, linear relationships, the authors also introduced NdSCA-simple, a computationally efficient version of the tool.

Validation and Key Results

The researchers validated NdSCA using synthetic data and two fMRI datasets (participants watching The Shawshank Redemption and various dance clips).

A key breakthrough is the ability to produce feature-specific connectivity maps. While traditional RRR produces a single "mixed" map, NdSCA reveals distinct "channels" the brain uses for different types of information. For instance, in the dance dataset, playing clips backward specifically disrupted motion-related communication between visual areas while leaving audio-related communication intact—a distinction invisible to standard methods.

Comparison of NdSCA with standard functional connectivity (FC), RRR, and PPI, demonstrating NdSCA's unique ability to separate audio and image-related interactions.

In the movie dataset, NdSCA successfully mapped face-related interactions to the Fusiform Face Area (FFA) and word-related interactions to the auditory cortex (A1) and language regions. Notably, it identified word-related information sharing where common alternatives like Psychophysiological Interaction (PPI) failed to find significant results.

Discussion and Future Directions

By accounting for spontaneous brain activity through a latent variable, NdSCA "cleans" neural data, providing a clearer view of stimulus-driven interactions. However, the method has higher data requirements due to its reliance on CVAEs.

Future work aims to extend the cross-areal mapping from linear to nonlinear models and apply NdSCA to high-temporal-resolution data like ECoG or MEG. This would allow researchers to observe how these demixed interactions evolve on a millisecond-by-millisecond basis, further deepening our understanding of the brain's complex communication networks.

Summary updated

Abstract

Multi-regional interaction among neuronal populations underlies the brain's processing of rich sensory information in our daily lives. Recent neuroscience and neuroimaging studies have increasingly used naturalistic stimuli and experimental design to identify such realistic sensory computation in the brain. However, existing methods for cross-areal interaction analysis with dimensionality reduction, such as reduced-rank regression and canonical correlation analysis, have limited applicability and interpretability in naturalistic settings because they usually do not appropriately 'demix' neural interactions into those associated with different types of task parameters or stimulus features (e.g., visual or audio). In this paper, we develop a new method for cross-areal interaction analysis that uses the rich task or stimulus parameters to reveal how and what types of information are shared by different neural populations. The proposed neural demixed shared component analysis combines existing dimensionality reduction methods with a practical neural network implementation of functional analysis of variance with latent variables, thereby efficiently demixing nonlinear effects of continuous and multimodal stimuli. We also propose a simplifying alternative under the assumptions of linear effects and unimodal stimuli. To demonstrate our methods, we analyzed two human neuroimaging datasets of participants watching naturalistic videos of movies and dance movements. The results demonstrate that our methods provide new insights into multi-regional interaction in the brain during naturalistic sensory inputs, which cannot be captured by conventional techniques.
BibTeX
@misc{takagi2021neural,
  author = {Yu Takagi and Laurence T. Hunt and Ryu Ohata and Hiroshi Imamizu and Jun-ichiro Hirayama},
  title = {{Neural dSCA: demixing multimodal interaction among brain areas during naturalistic experiments}},
  year = {2021},
  eprint = {2106.02948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.02948},
}
Figure