Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)

Yu Zhao, Xiang Li, Wei Zhang, Shijie Zhao, Milad Makkie, Mo Zhang, Quanzheng Li, Tianming Liu

TL;DR

Researchers have developed a Spatio-Temporal Convolutional Neural Network (ST-CNN) that jointly learns the spatial and temporal patterns of functional brain networks from 4D fMRI data. By integrating a 3D U-Net with a 1D Convolutional Auto-Encoder, the model outperforms traditional methods in identifying the Default Mode Network (DMN) and demonstrates remarkable generalizability across different populations and cognitive tasks.

What Problem Does This Paper Address?

Functional Magnetic Resonance Imaging (fMRI) captures the brain's activity by recording 3D volumes over a period of time, resulting in complex 4D datasets. A major goal in cognitive neuroscience is to identify "functional networks"—groups of brain regions that work together.

Traditionally, researchers have used methods like Independent Component Analysis (ICA) or sparse representation (SR) to find these networks. However, these methods often have a "blind spot": they typically prioritize either the spatial domain or the temporal domain, regressing the other out later. Because the spatial and temporal characteristics of brain activity are intrinsically intertwined, analyzing them separately can lead to inaccurate identifications, especially in subjects where standard assumptions (like "sparsity") don't hold.

Methods

The authors propose the ST-CNN, a framework designed to model both dimensions simultaneously. The architecture consists of three main components:

  1. Spatial Network (3D U-Net): Inspired by the popular U-Net used in medical image segmentation, this component takes 4D fMRI data and maps the 3D spatial volumes to a targeted functional network map (specifically the DMN).
  2. Combination Joint Operator: This is the "glue" of the model. It uses the 3D spatial map generated by the U-Net as a convolutional kernel. It slides this kernel over the original 4D fMRI data to extract a 1D time series that represents the network's activity over time.
  3. Temporal Network (1D CAE): A 1D Convolutional Auto-Encoder then models this extracted time series to capture the temporal dynamics of the network.

The training happens in three stages: first training the spatial network, then the temporal network, and finally fine-tuning the entire integrated system using a weighted loss function (10:1 spatial to temporal ratio).

Datasets

The study utilized high-quality data from the Human Connectome Project (HCP):

  • HCP Q1 Release: Used for training (52 subjects, motor task) and testing (13 subjects, motor task; 67 subjects, emotion task).
  • HCP S900 Release: Used for testing generalizability (100 randomly selected subjects, motor task).

The researchers used motor task data for training but tested the model on both motor and emotion tasks to see if the model could identify the DMN regardless of what the subject was actually doing in the scanner.

Key Insights and Ideas

  • Joint Learning is Key: The "aha moment" here is that by forcing the spatial and temporal networks to talk to each other during training, the model learns the intrinsic relationship between where a network is and how it behaves over time.
  • Moving Beyond Sparsity: Traditional sparse representation (SR) assumes that brain networks are spatially sparse. While often true, this prior can fail. The ST-CNN proved more robust because it learns features directly from the data rather than relying on a rigid mathematical assumption.
  • Cross-Task Stability: The DMN is a "background" network that should be present regardless of the specific task. The fact that an ST-CNN trained on a motor task could accurately find the DMN in an emotion task suggests the model is capturing fundamental biological signatures rather than task-specific noise.

Key Results

  • Superior Accuracy: In the HCP Q1 motor dataset, ST-CNN achieved a spatial overlap (Jaccard similarity) of 0.172 with the DMN template, significantly higher than the 0.115 achieved by sparse representation.
  • Robustness: The authors found that in 20% of cases (13 out of 65 subjects) in the HCP Q1 dataset, traditional sparse representation failed to identify the DMN entirely, while the ST-CNN succeeded.
  • Generalizability: The model maintained high performance when applied to the much larger S900 dataset and across different cognitive tasks (emotion vs. motor), proving it isn't "overfit" to a specific group of people or a specific experiment.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Targeted Networks: This study focused on the Default Mode Network because it is highly stable. Future work will need to extend this to task-related networks that are only present during specific activities.
  • Training Labels: The model currently uses results from sparse representation as "ground truth" for training. The authors suggest refining the framework by training it on networks identified by other established methods, like temporal ICA.
  • Clinical Application: The ultimate test will be applying ST-CNN to "abnormal" brains (e.g., patients with Alzheimer’s or Mild Cognitive Impairment) to see if it can characterize how these spatio-temporal patterns break down in disease.

Summary updated

Abstract

Simultaneous modeling of the spatio-temporal variation patterns of brain functional network from 4D fMRI data has been an important yet challenging problem for the field of cognitive neuroscience and medical image analysis. Inspired by the recent success in applying deep learning for functional brain decoding and encoding, in this work we propose a spatio-temporal convolutional neural network (ST-CNN) to jointly learn the spatial and temporal patterns of targeted network from the training data and perform automatic, pin-pointing functional network identification. The proposed ST-CNN is evaluated by the task of identifying the Default Mode Network (DMN) from fMRI data. Results show that while the framework is only trained on one fMRI dataset, it has the sufficient generalizability to identify the DMN from different populations of data as well as different cognitive tasks. Further investigation into the results show that the superior performance of ST-CNN is driven by the jointly-learning scheme, which capture the intrinsic relationship between the spatial and temporal characteristic of DMN and ensures the accurate identification.
BibTeX
@inproceedings{zhao2018modeling,
  author = {Yu Zhao and Xiang Li and Wei Zhang and Shijie Zhao and Milad Makkie and Mo Zhang and Quanzheng Li and Tianming Liu},
  title = {{Modeling 4D fMRI Data via Spatio-Temporal Convolutional Neural Networks (ST-CNN)}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  doi = {10.1007/978-3-030-00931-1\_21},
}

Loading related content...

Figure