Gray Matters: An Efficient Vision Transformer GAN Framework for Predicting Functional Network Connectivity Biomarkers from Brain Structure

Yuda Bi, Anees Abrol, Sihan Jia, Zening Fu, Vince D. CalhounbioRxiv 2024 · DOI

TL;DR

This paper introduces cEViT-GAN, a novel deep learning framework that uses Vision Transformers (ViTs) to generate functional brain connectivity (FNC) maps directly from structural MRI (sMRI) data. This approach aims to bridge the gap between brain structure and function, offering a more precise way to identify biomarkers for neurological disorders like schizophrenia.

What Problem Does This Paper Address?

Understanding the intricate relationship between brain structure and function is crucial for diagnosing and treating neurological disorders. While structural MRI (sMRI) provides detailed anatomical information and functional MRI (fMRI) captures brain activity, directly linking these two modalities to predict functional connectivity (FNC) from structure alone has been challenging. Existing methods often struggle with the complexity and non-linearity of these relationships. The paper addresses this by proposing a novel generative model that can learn these complex structure-function mappings, specifically aiming to identify biomarkers for schizophrenia.

Methods

The core of this work is the cEViT-GAN (conditional Efficient Vision Transformer Generative Adversarial Network). This framework leverages the power of Vision Transformers (ViTs) within a GAN architecture. Key innovations include:

  • 3D Patch Embedding: sMRI data is broken down into 3D patches, which are then processed through pre-trained ViT embeddings. This allows the model to capture rich spatial hierarchies from the structural data.
  • Efficient Block-wise Multi-head Self-Attention (BMHSA): To overcome the computational burden of standard ViT self-attention, especially with small image patches, the authors introduce BMHSA. This mechanism partitions computations into blocks, allowing for more efficient and localized attention while still capturing long-range dependencies.
  • Conditional Generation: The GAN is conditioned on subject labels (e.g., Schizophrenia (SZ) vs. Healthy Control (HC)), enabling it to generate FNC maps that reflect group-specific differences.
  • Generator and Discriminator Architecture: Both the generator and discriminator utilize efficient ViT architectures. The generator reconstructs FNC patches from embeddings, and the discriminator distinguishes between real and generated FNC maps.

Datasets

The study utilized two comprehensive datasets for clinical schizophrenia research:

  • Dataset 1: Aggregated data from three studies (fBIRN, MPRC, and COBRE) comprising 827 participants (477 controls, 350 with schizophrenia). Data was acquired using standard rsfMRI sequences across multiple sites and scanners.
  • Dataset 2: Data from 815 participants (326 controls, 489 with schizophrenia) collected from seven Chinese hospitals.

The study also mentions using pre-trained weights from a multimodal deep learning model developed in prior work for schizophrenia prediction, which was used for the 3D patch embedding layers.

Key Insights and Ideas

  • Structure Holds Significant Functional Information: The results strongly suggest that structural MRI data contains a substantial amount of information predictive of functional brain connectivity, even for complex disorders like schizophrenia.
  • ViTs are Powerful for Neuroimaging: The paper demonstrates the effectiveness of Vision Transformers in capturing intricate patterns within 3D brain imaging data, outperforming traditional CNN-based approaches in certain aspects.
  • Efficiency is Key for 3D Data: The development of the BMHSA mechanism highlights the critical need for efficient attention mechanisms when dealing with high-dimensional 3D neuroimaging data.
  • Bridging the Structure-Function Gap: The cEViT-GAN framework offers a promising avenue for directly linking structural brain features to functional connectivity patterns, potentially revolutionizing diagnostic approaches.

Key Results

  • High Correlation for Average FNC: The model achieved a strong correlation (0.97) between generated and actual whole-average FNC maps across all subjects.
  • Good Group Difference Correlation: The generated group-difference FNC (HC-SZ) showed a significant correlation (0.74) with real data, indicating the model's ability to capture group-specific functional alterations.
  • Biomarker Identification: Attention maps generated by the ViT highlighted key brain regions (mPFC, DL-PFC, temporal lobe, cerebellum) strongly associated with schizophrenia, aligning with existing neuroscientific knowledge.
  • Subcortical Structure Importance: The model demonstrated high fidelity in replicating FNC patterns related to subcortical structures, suggesting their critical role in differentiating SZ patients from controls.
  • Cross-domain Analysis: The study found a significant relationship (0.74) between structural and functional FNC matrices, though structural data alone did not fully capture all functional features, indicating the complexity of brain functionality.

Limitations and Future Work

  • Reduced Similarity in Group Differences: While the model performed well on average FNC, the similarity for group-difference FNCs was lower. This suggests that capturing nuanced, individual-specific functional information for discrimination is more challenging and warrants further investigation.
  • Future Exploration: The authors suggest future work should focus on improving the model's ability to capture these nuanced group differences, potentially leading to more precise diagnostic tools and personalized therapy. They also highlight the need for further research into the multidimensional nature of brain structure-function interactions.

Summary updated

Abstract

Abstract The field of brain connectivity research has under-gone revolutionary changes thanks to state-of-the-art advancements in neuroimaging, particularly regarding structural and functional magnetic resonance imaging (MRI). To navigate the intricate neural dynamics, one must possess a keen comprehension of the interdependent links between structure and function. Such relationships are understudied as they are complex and likely nonlinear. To address this, we created a new generative deep learning architecture using a conditional efficient vision transformer generative adversarial network (cEViTGAN) to capture the distinct information in structural and functional MRI of the human brain. Our model generates functional network connectivity (FNC) matrices directly from three-dimensional sMRI data. Two pioneering innovations are central to our approach. First, we use a novel linear embedding method for structural MRI (sMRI) data that retains the 3D spatial detail. This embedding is best for representative learning, and when used on a consistent dataset, and shows that it is good at upstream classification assignments. To estimate neural biomarkers, we need to process much smaller patches using ViT-based architectures, which usually makes the computations more difficult because of the self-attention operations. We present a new, lightweight self-attention mechanism to address this challenge. Our mechanism not only overcomes computational shortcomings of traditional softmax self-attention but also surpasses pure linear self-attention models in accuracy and performance. This optimization enables us to analyze even the tiniest neuroanatomical details with exceptional precision. Our model allows for the identification of functional network connectivity (FNC) with 74.2% accuracy and also predicts subject differences in FNC for schizophrenia patients versus controls. The results are intriguing and suggest the links between gray matter volume and brain function may be stronger than previously considered.
BibTeX
@misc{bi2024gray,
  author = {Yuda Bi and Anees Abrol and Sihan Jia and Zening Fu and Vince D. Calhoun},
  title = {{Gray Matters: An Efficient Vision Transformer GAN Framework for Predicting Functional Network Connectivity Biomarkers from Brain Structure}},
  year = {2024},
  doi = {10.1101/2024.01.11.575307},
  howpublished = {bioRxiv},
}
Figure