Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)

Yu Zhao, Xiang Li, Wei Zhang, Shijie Zhao, Milad Makkie, Mo Zhang, Quanzheng Li, Tianming Liu

Abstract

Simultaneous modeling of the spatio-temporal variation patterns of brain functional network from 4D fMRI data has been an important yet challenging problem for the field of cognitive neuroscience and medical image analysis. Inspired by the recent success in applying deep learning for functional brain decoding and encoding, in this work we propose a spatio-temporal convolutional neural network (ST-CNN) to jointly learn the spatial and temporal patterns of targeted network from the training data and perform automatic, pin-pointing functional network identification. The proposed ST-CNN is evaluated by the task of identifying the Default Mode Network (DMN) from fMRI data. Results show that while the framework is only trained on one fMRI dataset, it has the sufficient generalizability to identify the DMN from different populations of data as well as different cognitive tasks. Further investigation into the results show that the superior performance of ST-CNN is driven by the jointly-learning scheme, which capture the intrinsic relationship between the spatial and temporal characteristic of DMN and ensures the accurate identification.

Introduction

Functional Magnetic Resonance Imaging (fMRI) is a cornerstone of modern cognitive neuroscience, capturing brain activity as a series of 3D volumes over time. This 4D data structure is rich but complex, making the identification of functional networks—sets of brain regions exhibiting synchronized activity—a significant challenge. Traditional approaches, such as Independent Component Analysis (ICA) and sparse representation (SR), often treat spatial and temporal domains sequentially. By prioritizing one domain and regressing out the other, these methods may overlook the intrinsic spatio-temporal dependencies of brain activity, leading to suboptimal network identification, particularly when standard assumptions like spatial sparsity are violated.

Methods

To address these limitations, we introduce the Spatio-Temporal Convolutional Neural Network (ST-CNN). This framework is designed to model spatial and temporal dimensions simultaneously through a multi-component architecture.

Architecture

The ST-CNN consists of three primary modules:

  1. Spatial Network (3D U-Net): This component maps 4D fMRI input to a targeted 3D functional network map, such as the Default Mode Network (DMN).
  2. Combination Joint Operator: This operator uses the 3D spatial map produced by the U-Net as a convolutional kernel. By sliding this kernel over the original 4D fMRI data, it extracts a 1D time series representing the network's temporal activity.
  3. Temporal Network (1D CAE): A 1D Convolutional Auto-Encoder models the extracted time series to capture the network's underlying temporal dynamics.
The ST-CNN architecture (a) and detailed components (b), including the 3D U-Net for spatial mapping and the 1D CAE for temporal modeling.

Training and Evaluation

The model is trained in three stages: initial spatial network training, followed by temporal network training, and finally joint fine-tuning of the integrated system using a weighted loss function (10:1 spatial-to-temporal ratio). Evaluation was conducted using high-quality data from the Human Connectome Project (HCP), specifically the Q1 release (motor and emotion tasks) and the S900 release for generalizability testing.

Results

The ST-CNN demonstrated superior performance and robustness compared to traditional methods. In the HCP Q1 motor dataset, the model achieved a Jaccard similarity of 0.172 with the DMN template, outperforming sparse representation (0.115). Notably, ST-CNN successfully identified the DMN in 20% of subjects where traditional SR failed entirely.

Qualitative comparison of ground-truth and ST-CNN predicted spatial maps and temporal time series for HCP Q1 and S900 datasets.

The model also exhibited remarkable generalizability. Despite being trained on motor task data, it accurately identified the DMN in subjects performing emotion tasks. This stability across different cognitive states suggests that ST-CNN captures fundamental biological signatures of functional networks rather than task-specific noise.

Discussion and Conclusion

The success of ST-CNN underscores the importance of joint spatio-temporal learning. By forcing the spatial and temporal modules to interact, the model learns the intrinsic relationship between a network's location and its behavior over time. This data-driven approach proves more robust than methods relying on rigid mathematical priors like sparsity.

Future research will focus on extending this framework to task-related networks that are more transient than the DMN. Additionally, exploring clinical applications—such as identifying altered spatio-temporal patterns in patients with Alzheimer’s disease—remains a critical next step for validating the utility of ST-CNN in medical diagnostics.

Summary updated

Abstract

Simultaneous modeling of the spatio-temporal variation patterns of brain functional network from 4D fMRI data has been an important yet challenging problem for the field of cognitive neuroscience and medical image analysis. Inspired by the recent success in applying deep learning for functional brain decoding and encoding, in this work we propose a spatio-temporal convolutional neural network (ST-CNN)to jointly learn the spatial and temporal patterns of targeted network from the training data and perform automatic, pin-pointing functional network identification. The proposed ST-CNN is evaluated by the task of identifying the Default Mode Network (DMN) from fMRI data. Results show that while the framework is only trained on one fMRI dataset,it has the sufficient generalizability to identify the DMN from different populations of data as well as different cognitive tasks. Further investigation into the results show that the superior performance of ST-CNN is driven by the jointly-learning scheme, which capture the intrinsic relationship between the spatial and temporal characteristic of DMN and ensures the accurate identification.
BibTeX
@inproceedings{zhao2018modeling,
  author = {Yu Zhao and Xiang Li and Wei Zhang and Shijie Zhao and Milad Makkie and Mo Zhang and Quanzheng Li and Tianming Liu},
  title = {{Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  doi = {10.1007/978-3-030-00931-1\_21},
}
Figure