Multi-modal deep learning from imaging genomic data for schizophrenia classification

Ayush Kanyal, Badhan Mazumder, Vince D. Calhoun, Adrian Preda, Jessica Turner, Judith M. Ford, Dong Hye YeFrontiers in Psychiatry 2024 · DOI

TL;DR

Researchers have developed a multi-modal deep learning framework that integrates brain structure (sMRI), function (fMRI), and genetics (SNPs) to classify schizophrenia with 79% accuracy. By leveraging Explainable AI (XAI), the team moved beyond "black box" predictions to identify the specific neural connections and genetic markers that drive the disorder.

What Problem Does This Paper Address?

Schizophrenia (SZ) is a notoriously complex neuropsychiatric disorder. Its origins aren't tied to a single "broken" part of the brain; instead, it involves a messy interplay of structural changes, functional "disconnections," and hereditary genetic factors.

Most existing diagnostic research suffers from two main issues:

  1. Modality Silos: Studies often look at only one type of data (e.g., just MRI scans), missing the synergistic information found when combining imaging with genomics.
  2. The Black Box Problem: Deep learning models are often accurate but inscrutable. In a clinical setting, a "yes/no" diagnosis without an explanation of why is of limited use to doctors.

This paper tackles these gaps by fusing three distinct data types and using Explainable AI to pinpoint the biological biomarkers behind the classification.

Methods

The authors proposed a two-stage pipeline that combines feature extraction, feature selection via XAI, and a final tree-based classifier.

  • Structural MRI (sMRI): They used a 3D-modified DenseNet121 (pre-trained) to extract 1,024 morphological features, such as gray matter volume, from brain scans.
  • Functional Network Connectivity (fMRI): To handle the 1,378 possible connections in the brain's functional network, they trained a 1D-Convolutional Neural Network (CNN). They then applied Layer-wise Relevance Propagation (LRP)—an XAI technique—to identify and keep only the top 249 most relevant connections.
  • Genomics (SNPs): Dealing with nearly 5,000 Single Nucleotide Polymorphisms (SNPs) is a "needle in a haystack" problem. They used another 1D-CNN combined with LRP to prune the list down to the 100 most significant genetic markers.
  • Fusion & Classification: These features were concatenated into a single vector and fed into an XGBoost classifier, a powerful gradient-boosting algorithm known for handling tabular, multi-modal data efficiently.

Datasets

The study utilized a robust combined dataset from three major research cohorts:

  • FBIRN (Functional Biomedical Informatics Research Network)
  • BSNIP (Bipolar and Schizophrenia Network on Intermediate Phenotypes)
  • COBRE (Center for Biomedical Research Excellence)

In total, the study included 492 participants (219 diagnosed with SZ and 273 healthy controls), providing a more diverse and representative sample than many single-site studies.

Key Insights and Ideas

  • The Power of "And": The most striking insight is the synergy between modalities. While fMRI was the strongest single predictor, adding structural and genetic data provided a significant performance boost, proving that these data types contain non-overlapping, complementary information.
  • XAI as a Filter: Instead of just using XAI to explain the final result, the authors used it as a sophisticated feature selection tool. By using LRP to "see" which features the CNN found important, they could discard noise (irrelevant SNPs or brain connections) before the final classification.
  • Biological Grounding: The model didn't just find random patterns; the XAI highlighted disruptions in the subcortical (SC) and sensorimotor (SM) regions. This aligns perfectly with the clinical "disconnection hypothesis," which suggests SZ is characterized by irregular neural connectivity.

Key Results

The multi-modal approach outperformed all single-modality benchmarks:

  • Multi-modal (sMRI + fMRI + SNP): 79.01% Accuracy
  • fMRI only: 75.29% Accuracy
  • sMRI only: 66.36% Accuracy
  • SNP (Genetics) only: 57.06% Accuracy

The framework achieved a balanced performance across the board, with a Precision of 78.83% and a Recall of 79.43%, suggesting it is equally good at identifying SZ cases and avoiding false positives.

Limitations and Future Work

While promising, the authors acknowledge several hurdles:

  • Data Standardization: MRI data can vary wildly depending on the scanner used. Future work needs to implement better standardization techniques to ensure the model isn't accidentally learning "scanner noise" instead of biological signals.
  • Sample Size: While 492 subjects is a good start for neuroimaging, deep learning typically thrives on thousands of samples. Expanding the dataset will be crucial for generalizability.
  • Clinical Integration: The current model performs binary classification (SZ vs. Healthy). Real-world clinical utility would require the model to differentiate SZ from other similar disorders, like bipolar disorder or schizoaffective disorder.

Summary updated

Abstract

We proposed a deep learning based framework that selects multi-modal (sMRI, fMRI and genetic) features efficiently and fuse them to obtain improved classification scores. Additionally, by using Explainable AI (XAI), we were able to pinpoint and validate significant functional network connections and SNPs that contributed the most toward SZ classification, providing necessary interpretation behind our findings.
BibTeX
@inproceedings{kanyal2024multi,
  author = {Ayush Kanyal and Badhan Mazumder and Vince D. Calhoun and Adrian Preda and Jessica Turner and Judith M. Ford and Dong Hye Ye},
  title = {{Multi-modal deep learning from imaging genomic data for schizophrenia classification}},
  booktitle = {Frontiers in Psychiatry},
  year = {2024},
  doi = {10.3389/fpsyt.2024.1384842},
}
Figure