Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders

Md Abdur Rahaman, Yash Garg, Armin Iraji, Zening Fu, Peter Kochunov, L. Elliot Hong, Theo G.m. van Erp, Adrian Preda, Jiayu Chen, Vince D. CalhounbioRxiv 2024 · DOI

TL;DR

This paper introduces a novel multi-modal fusion framework that combines brain imaging (structural and functional MRI) and genetic data to improve the classification of schizophrenia. Their key innovation is a "spatio-modality attention" module that intelligently weighs information from different data sources and spatial locations, leading to a highly accurate and interpretable model.

What Problem Does This Paper Address?

Studying complex neuropsychiatric disorders like schizophrenia is challenging because they affect multiple biological systems, each providing a unique but often disparate view of the underlying pathology. Integrating these diverse data sources (e.g., brain scans and genetic information) is difficult due to differences in data representation, noise, and high dimensionality. Existing methods struggle to effectively combine these modalities, leading to incomplete understanding and suboptimal diagnostic accuracy. This paper aims to bridge this gap by developing a sophisticated fusion technique that can harness the complementary strengths of imaging and genetic data for more accurate and interpretable schizophrenia detection.

Methods

The authors propose a multi-modal deep learning framework. The core innovation is a spatio-modality attention module. This module operates on latent representations extracted from modality-specific deep neural networks (an autoencoder for functional connectivity, an MLP for structural MRI loadings, and a bi-LSTM with attention for genetic data).

The attention module has two branches:

  1. Modality Attention: This branch learns to assign importance scores to each data modality, identifying which source is most informative for classification.
  2. Spatial Attention: This branch uses dilated convolutional neural networks to capture contextual patterns across the brain's spatial dimensions. Dilated convolutions are key here, as they allow for a larger receptive field without increasing computational cost, enabling the model to consider broader spatial relationships.

These two attention mechanisms are merged to create a spatio-modality attention mask. This mask is then used to weight the fused features, emphasizing the most relevant information. The framework is trained end-to-end, allowing the entire system to learn optimal fusion strategies. A significant aspect is the model's inherent interpretability, provided by the attention scores, which highlight important features and their contributions to the classification decision.

Datasets

The study utilizes a combined dataset of 437 subjects, comprising 162 individuals with schizophrenia (SZ) and 275 healthy controls (HC). This dataset is an aggregation of three publicly available neuroimaging-genetics datasets: COBRE, fBIRN, and MPRC. The paper also includes experiments on subsets of this data and random splits to assess reproducibility.

Key Insights and Ideas

  • Synergy of Modalities: Combining structural MRI, functional MRI (specifically static functional connectivity), and genetic data significantly outperforms unimodal approaches for schizophrenia classification.
  • The Power of Dilated Convolutions: Using dilated convolutions in the spatial attention branch is crucial for capturing contextual information from brain imaging data, leading to better performance than standard convolutions or fully connected layers for spatial feature extraction.
  • Interpretability through Attention: The spatio-modality attention scores provide a powerful tool for understanding why the model makes a particular classification. This allows researchers to identify specific brain regions, functional connections, and even genes that are most indicative of schizophrenia.
  • Key Brain Regions Identified: The model highlights the importance of structural components like the supplementary motor area (SMA), caudate, and insula, as well as functional connectivity within subcortical, sensorimotor, and cognitive control domains, in characterizing schizophrenia.
  • Genetic Contribution: While genetic data alone is insufficient, it plays a crucial complementary role when fused with imaging data, suggesting complex gene-environment interactions in the disorder.

Key Results

  • The proposed multi-modal spatio-modality attentive fusion model achieved an accuracy of 94.10% (P < 0.0001) for schizophrenia classification, outperforming state-of-the-art unimodal and multi-modal models.
  • The F1 score was 0.697, demonstrating a good balance between precision and recall, especially considering the imbalanced nature of the dataset.
  • The model's attention analysis identified specific brain regions (SMA, caudate, insula, temporal pole) and functional connections (among sensorimotor, subcortical, and cognitive control domains) as highly relevant for schizophrenia.
  • Biclustering of attention scores revealed a multi-modal cluster including genes CSMD1, ATK3, MOB4, and HSPE1, all previously linked to schizophrenia.

Limitations and Future Work

  • Dataset Size and Imbalance: While the combined dataset is substantial, the authors acknowledge the dataset is slightly imbalanced, which they addressed using the F1 score. Future work could explore more advanced techniques for handling class imbalance.
  • Generalizability: Although reproducibility was tested across different subsets and random splits, further validation on larger, more diverse datasets would strengthen the generalizability of the findings.
  • Modality Reduction: The paper mentions that a reduction ratio in the modality direction could help in combining a larger number of modalities in future studies.
  • Explaining Complex Interactions: While the model identifies key features, further research is needed to fully elucidate the complex interplay between genomics, structural, and functional brain alterations in schizophrenia. The authors suggest that tri-modal feature sets can potentially explain behavioral deficits from multiple biological perspectives.
  • Clinical Applications: The interpretability of the model opens avenues for exploring its potential in clinical settings for diagnosis and understanding disease mechanisms.

Summary updated

Abstract

Abstract Multi-modal learning has emerged as a powerful technique that leverages diverse data sources to enhance learning and decision-making processes. Adapting this approach to analyzing data collected from different biological domains is intuitive, especially for studying neuropsychiatric disorders. A complex neuropsychiatric disorder like schizophrenia (SZ) can affect multiple aspects of the brain and biologies. These biological sources each present distinct yet correlated expressions of subjects’ underlying physiological processes. Joint learning from these data sources can improve our understanding of the disorder. However, combining these biological sources is challenging for several reasons: (i) observations are domains-specific, leading to data being represented in dissimilar subspaces, and (ii) fused data is often noisy and high-dimensional, making it challenging to identify relevant information. To address these challenges, we propose a multi-modal artificial intelligence (AI) model with a novel fusion module inspired by a bottleneck attention module (BAM). We use deep neural networks (DNN) to learn latent space representations of the input streams. Next, we introduce a two-dimensional (spatio-modality) attention module to regulate the intermediate fusion for SZ classification. We implement spatial attention via a dilated convolutional neural network that creates large receptive fields for extracting significant contextual patterns. The resulting joint learning framework maximizes complementarity allowing us to explore the correspondence among the modalities. We test our model on a multi-modal imaging-genetic dataset and achieve an SZ prediction accuracy of 94.10% (P &lt; 0.0001), outperforming state-of-the-art unimodal and multi-modal models for the task. Moreover, the model provides inherent interpretability that helps identify concepts significant for the neural network’s decision and explains the underlying physiopathology of the disorder. Results also show that functional connectivity among subcortical, sensorimotor, and cognitive control domains plays an important role in characterizing SZ. Analysis of the spatio-modality attention scores suggests that structural components like the supplementary motor area, caudate, and insula play a significant role in SZ. Biclustering the attention scores discover a multi-modal cluster that includes genes CSMD1, ATK3, MOB4, and HSPE1, all of which have been identified as relevant to schizophrenia. In summary, feature attribution appears to be especially useful for probing the transient and confined but decisive patterns of complex disorders, and it shows promise for extensive applicability in future studies.
BibTeX
@misc{rahaman2024multi,
  author = {Md Abdur Rahaman and Yash Garg and Armin Iraji and Zening Fu and Peter Kochunov and L. Elliot Hong and Theo G.M. van Erp and Adrian Preda and Jiayu Chen and Vince D. Calhoun},
  title = {{Multi-modal Spatial-modality Attentive Fusion for Studying Neuropsychiatric Disorders}},
  year = {2024},
  doi = {10.1101/2024.06.09.598091},
  howpublished = {bioRxiv},
}
Figure