st-DenseViT: A Weakly Supervised Spatiotemporal Vision Transformer for Dense Prediction of Dynamic Brain Networks

Behnam Kazemivash, Pranav Suresh, Dong Hye Ye, Armin Iraji, Jingyu Liu, Sergey Plis, Peter Kochunov, Vince D. CalhounbioRxiv 2024 · DOI

Abstract

This work presents an important new approach for dynamic brain mapping, potentially opening up new opportunities for studying brain dynamics within specific networks. By framing the problem as a spatiotemporal dense prediction task in computer vision, we leverage the spatiotemporal ViT architecture combined with weakly supervised learning techniques to efficiently and effectively estimate these maps.

Introduction

Functional MRI (fMRI) is a cornerstone of modern neuroscience, yet capturing the simultaneous spatial and temporal evolution of complex neural networks remains a significant challenge. Traditional methods often rely on static spatial parcels, failing to account for the dynamic shifts in network shape and intensity over time. Furthermore, the lack of high-resolution "ground truth" labels for every millisecond of brain activity makes standard supervised deep learning difficult to apply.

To address these limitations, we introduce st-DenseViT, a transformer-based approach that treats brain mapping as a spatiotemporal dense prediction task. By utilizing weakly supervised learning, st-DenseViT generates personalized 4D dynamic brain networks, providing a high-resolution view of the brain's internal choreography.

Methodology

The st-DenseViT architecture is designed to handle the high-dimensional nature of fMRI data by framing it as a computer vision problem.

The st-DenseViT architecture overview, illustrating the fMRI input processing, the encoder-decoder pipeline, and the two proposed encoder configurations: Space-Time and Sequential.

The model consists of three primary components:

  1. Spatiotemporal Encoder: We evaluated two configurations based on the Vision Transformer (ViT). The Space-Time Encoder jointly processes spatial and temporal tokens to capture global context, while the Sequential Encoder processes temporal patterns before spatial ones.
  2. CNN Decoder: This component uses a series of 3D transposed convolutions to project abstract transformer features back into the original 3D brain volume, enabling voxel-wise (dense) prediction.
  3. Weak Supervision: Training is guided by spatially constrained windowed Independent Component Analysis (ICA). While ICA provides a noisy estimate of brain networks, st-DenseViT learns to "clean up" and interpolate this data, effectively acting as a denoiser.

The training process employs a custom loss function combining robust regression (log-cosh) with a Structural Similarity Index (SSIM) loss to ensure both numerical accuracy and structural coherence in the generated maps.

Data and Experiments

The study utilized resting-state fMRI data from three major repositories: MPRC, FBIRN, and COBRE, totaling 508 datasets. The population included 315 Healthy Controls (HC) and 193 individuals diagnosed with Schizophrenia (SCZ). All data were aligned using the NeuroMark template to ensure reproducible results across subjects.

Results and Discussion

Our experiments revealed several key insights:

  • Denoising Capabilities: st-DenseViT consistently outperformed its "weak" ICA teacher, producing smoother and more biologically plausible transitions in brain activity.
  • High Fidelity Reconstruction: The model achieved high accuracy across various networks. For instance, the Cerebellar network was reconstructed with a Mean Absolute Relative Error of 0.10, while the Default Mode Network (DMN) achieved a high SSIM of 0.87.
  • Clinical Biomarkers: The model identified significant activity differences in the thalamus within the DMN, where healthy controls exhibited higher activity compared to schizophrenia patients. This finding aligns with existing clinical literature but provides unprecedented spatiotemporal detail.
  • Dynamic Signatures: By analyzing "temporal gradient maps," we found statistically significant differences between HC and SCZ groups, suggesting that st-DenseViT can identify dynamic biomarkers for mental health conditions.

Conclusion

st-DenseViT represents a significant step forward in dynamic neuroimaging. By leveraging the power of spatiotemporal transformers and weak supervision, it enables the creation of high-resolution, personalized 4D brain maps. Future work will focus on increasing temporal resolution and exploring self-supervised techniques that reduce dependency on ICA templates. Additionally, applying this framework to other neurological conditions like Alzheimer's or Bipolar disorder could uncover unique dynamic signatures of pathology.

Summary updated

Abstract

This work presents an important new approach for dynamic brain mapping, potentially opening up new opportunities for studying brain dynamics within specific networks. By framing the problem as a spatiotemporal dense prediction task in computer vision, we leverage the spatiotemporal ViT architecture combined with weakly supervised learning techniques to efficiently and effectively estimate these maps.
BibTeX
@misc{kazemivash2024st,
  author = {Behnam Kazemivash and Pranav Suresh and Dong Hye Ye and Armin Iraji and Jingyu Liu and Sergey Plis and Peter Kochunov and Vince D. Calhoun},
  title = {{st-DenseViT: A Weakly Supervised Spatiotemporal Vision Transformer for Dense Prediction of Dynamic Brain Networks}},
  year = {2024},
  doi = {10.1101/2024.11.28.625914},
  howpublished = {bioRxiv},
}
Figure