Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data

Mohammad Hosseini, Maryam M. Shanechi

Introduction

Modern neuroimaging techniques, such as widefield calcium imaging (WFCI) and functional ultrasound imaging (fUSI), provide an unprecedented view of brain activity across large spatial scales. However, the analysis of these data streams is complicated by high dimensionality, complex spatiotemporal dependencies, and the presence of significant neural activity that is unrelated to the specific behaviors under study. Traditional modeling approaches typically address these challenges through "lossy" preprocessing steps, such as Principal Component Analysis (PCA) or the use of pre-defined Regions of Interest (ROIs). While these methods reduce dimensionality, they often discard subtle spatiotemporal patterns that are essential for understanding the link between brain dynamics and behavior.

The SBIND Framework

To overcome these limitations, the authors introduce SBIND (Spatiotemporal modeling of Behavior in Imaging Neural Data), a novel deep learning framework designed to learn directly from raw imaging data. SBIND avoids the pitfalls of manual preprocessing by utilizing an architecture that preserves the inherent spatial structure of the brain.

Architecture and Disentanglement

SBIND is built upon two primary pillars:

  1. Disentangled Two-Phase Learning: The framework employs two separate Convolutional Recurrent Neural Networks (ConvRNNs). In the first phase, the model is trained to identify "behaviorally relevant" latent states—patterns that directly predict the subject's actions. In the second phase, a separate network captures the "residual" dynamics, which represent structured neural activity that is independent of the studied behavior.
  2. ConvRNN with Self-Attention: To account for the brain's complex spatial dependencies, SBIND integrates convolutional layers for local feature extraction with a self-attention mechanism. This allows the model to capture long-range functional connectivity, learning how activity in one region relates to activity in distant areas of the brain.

Experimental Validation

The model was validated across three distinct and challenging datasets:

  • WFCI 1: Data from mice performing a decision-making task involving auditory and visual cues, featuring 14 dimensions of continuous behavior.
  • WFCI 2: A mouse dataset focused on categorical behavior derived from binary sensor traces.
  • fUSI: Functional ultrasound imaging from non-human primates performing memory-guided movements. This represents a significant milestone, as it is the first time dynamical latent modeling has been applied to the fUSI modality.

Key Insights and Results

SBIND demonstrates a clear advantage by processing raw data rather than pre-compressed features. Even when baseline models were provided with optimized preprocessed features, SBIND's ability to maintain spatial context resulted in superior performance.

Performance Metrics

  • Superior Decoding: SBIND consistently outperformed state-of-the-art models like CEBRA and DPAD. In the WFCI 1 dataset, SBIND achieved a behavior Mean Squared Error (MSE) of approximately 0.49, whereas baselines ranged from 0.59 to 0.62.
  • fUSI Breakthrough: The model achieved 73% accuracy in decoding movement directions from fUSI data, suggesting high potential for non-invasive Brain-Computer Interface (BCI) applications.
  • Global Context: Ablation studies and attribution maps revealed that the self-attention mechanism is critical. Without it, models suffer from "tunnel vision," failing to leverage the brain-wide information necessary for accurate prediction.
  • Real-Time Potential: Despite its architectural complexity, SBIND maintains high efficiency. A single inference step takes only 17.9 ms on a modern GPU, which is faster than the sampling rate of the imaging modalities themselves.

Discussion and Future Work

SBIND proves that "noise" in neuroimaging is often structured neural activity that simply requires a more sophisticated modeling approach to disentangle. By explicitly modeling behaviorally irrelevant dynamics, SBIND achieves higher accuracy in predicting future neural states.

However, several avenues for future research remain. Current models are largely session-specific; extending SBIND to handle multi-session or multi-animal data is a priority. Additionally, the model currently assumes a stable brain-behavior relationship, which could be improved by incorporating adaptive learning to account for non-stationarity during skill acquisition. Finally, scaling the self-attention mechanism to handle 3D volumes (such as fMRI) will require further computational optimization to manage the quadratic growth of attention maps.

Summary updated

BibTeX
@inproceedings{hosseini2025dynamical,
  author = {Mohammad Hosseini and Maryam M. Shanechi},
  title = {{Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=k4KVhQd19x},
}
Figure