TL;DR
This paper introduces MultiCrossViT, a novel multimodal deep learning model that combines structural MRI (sMRI) and functional network connectivity (sFNC) data to predict schizophrenia. The model achieves a promising AUC of 0.833, outperforming existing single-modality and multimodal approaches, and offers insights into brain regions associated with the disease through attention map visualization.
What Problem Does This Paper Address?
Schizophrenia is a severe mental disorder that affects perception, thought, and behavior. While functional MRI (fMRI) data has shown promise in identifying schizophrenia, its computational cost often limits its use in deep learning models. Structural MRI (sMRI) data is more readily available but has historically been less effective on its own for accurate classification. This paper aims to bridge this gap by developing a deep learning model that can effectively leverage both sMRI and static functional network connectivity (sFNC) data, a computationally less intensive representation derived from fMRI, to improve schizophrenia prediction.
Methods
The core of this work is the MultiCrossViT architecture, a multimodal vision transformer. It processes sMRI and sFNC data through separate input channels, employing 3D and 2D patch embedding respectively, followed by distinct transformer encoders. The key innovation lies in the cross-attention mechanism integrated after each transformer encoder layer. This mechanism allows the model to fuse information from both modalities, capturing their joint patterns, which is crucial for both prediction and generating interpretable saliency maps. Self-attention is used within each modality's encoder, while cross-attention facilitates inter-modality interaction.
Datasets
The study utilizes data from a multi-site schizophrenia study gathered by TRENDS. This dataset includes:
- Structural MRI (sMRI): 3D data with an initial shape of (121, 145, 121).
- Static Functional Network Connectivity (sFNC): 2D data derived from fMRI time series, represented as (53, 53) matrices.
The data was preprocessed and augmented using the Torchio library. The dataset consists of 2130 subjects.
Key Insights and Ideas
- Multimodal Fusion is Key: Combining sMRI and sFNC data significantly enhances schizophrenia prediction accuracy compared to using either modality alone.
- Cross-Attention for Effective Fusion: The proposed cross-attention mechanism is more effective at integrating information from different modalities than simple concatenation.
- Transformers for Medical Imaging: Vision Transformers, originally developed for natural images, show strong potential in analyzing complex medical imaging data like MRI.
- Interpretability through Attention: Visualizing the attention maps from the transformer encoders provides insights into which brain regions and functional networks are most relevant for distinguishing between healthy individuals and those with schizophrenia.
Key Results
The MultiCrossViT model achieved a notable AUC of 0.833, which is significantly higher than the average AUC of 0.766 for single-modality baselines and 0.78 for multimodal baselines. The paper also highlights that MultiCrossViT has fewer parameters than other multimodal architectures, suggesting a more efficient model. Saliency maps revealed that patients with schizophrenia showed increased activity in posterior temporal lobe and thalamus regions, and more salient intra-cerebellar connectivity, as well as auditory-motor and auditory-visual connectivity patterns.
Limitations and Future Work
- Dataset Size: While the study achieved good results on a dataset with a "minimal training subjects" (referring to the context of medical imaging, not necessarily small absolute numbers), the authors acknowledge that larger datasets could further improve performance.
- Generalizability: The study's findings are based on a specific dataset. Future work could explore the generalizability of MultiCrossViT to other independent datasets.
- Dynamic fMRI: The current model uses static FNC. Investigating the use of dynamic functional connectivity derived from fMRI could potentially capture more nuanced temporal brain activity patterns.
- Further Interpretability: While saliency maps are provided, further research could delve deeper into the specific patterns identified and their clinical implications.
Summary updated