Classification of schizophrenia spectrum disorder using machine learning and functional connectivity: reconsidering the clinical application

Chao Li, Fei Wang, Xiaowei Jiang, Ji Chen, Jia Duan, Shaoqiang Han, Hao Yan, Yanqing Tang, Ke XumedRxiv 2020 · DOI

Abstract

An accurate identification of schizophrenia spectrum disorder (SSD) at an early stage could potentially allow for treating SSD with appropriate intervention to potentially prevent future deterioration. Despite mounting studies finding that neuroimaging combined with machine learning can identify chronic medicated SSD, whether or not the classification model identifies the trait biomarker of SSD that can be used to identify early-stage SSD is largely unknown. The present study aimed to investigate whether or not the classification model trained using chronic medicated SSD identified the trait biomarker of SSD and whether the model can be generalized to early-stage SSD, by using functional connectivity (FC) combined with support vector machine (SVM) using a large sample from 4 independent sites (n = 1077). We found that the classification model trained using chronic medicated SSD from three sites (dataset 2, 3 and 4) classified SSD from HCs in another site (dataset 1) with 69% accuracy ($P = 2.86 \times 10^{-13}$). Subgroup analysis indicated that this model can identify chronic medicated SSD in dataset 1 with 71% sensitivity ($P = 4.63 \times 10^{-5}$), but cannot be generalized to first-episode unmedicated SSD (sensitivity = 48%, $P = 0.68$) and first-episode medicated SSD (sensitivity = 59%, $P = 0.10$). Univariable analysis showed that medication usage had a significant effect on FC, but disease duration had no significant effect on FC. These findings suggest that the classification model trained using chronic medicated SSD may mainly identify the pattern of chronic medication usage state, rather than the trait biomarker of SSD. Therefore, we should reconsider the current machine learning studies in chronic medicated SSD more cautiously in terms of clinical application.

Introduction: The Challenge of Early Diagnosis

The "holy grail" of psychiatric neuroimaging is the discovery of a clinically usable biomarker—an objective brain signature that can diagnose disorders like schizophrenia spectrum disorder (SSD) before symptoms become chronic. While many studies have successfully used machine learning (ML) and functional magnetic resonance imaging (fMRI) to distinguish patients from healthy controls with high accuracy, a significant caveat remains: most of these models are trained on "chronic medicated" patients.

This study addresses a critical gap in the literature: Generalizability. If a model is trained on patients who have been ill and medicated for years, can it still identify a patient in their first episode who hasn't started treatment? If the model fails, it suggests that what we thought were "disease markers" might actually be "medication markers" or signatures of chronicity.

Methods and Materials

The researchers employed a robust machine learning pipeline focused on Functional Connectivity (FC), which measures how different brain regions synchronize their activity during rest. The study utilized a large, diverse sample of 1,077 participants (502 SSD patients and 575 healthy controls) across four independent sites.

Workflow of the machine learning pipeline for SSD classification across multiple sites.

The Machine Learning Pipeline

As illustrated in Figure 1, the workflow involved several key stages:

  1. Preprocessing: Standard fMRI cleaning (motion correction, denoising) using SPM12 and DPABI.
  2. Feature Extraction: The Brainnetome atlas was used to divide the brain into 246 regions, calculating the Pearson correlation between every pair to generate over 30,000 unique connectivity features per subject.
  3. Dimensionality Reduction: Principal Component Analysis (PCA) was used to compress these features, retaining 95% of the variance.
  4. Classification: A Linear Support Vector Machine (SVM) was trained on data from three sites (Datasets 2, 3, and 4) and then tested on a completely independent fourth site (Dataset 1) to ensure the results were not overfitted to a specific scanner.
  5. Subgroup Analysis: This was the study's key innovation. The trained model was tested specifically on three groups within the test site: chronic medicated patients, first-episode medicated patients, and first-episode unmedicated patients.

Key Results: The Generalization Gap

The model's performance revealed a stark contrast between patient subgroups, highlighting the influence of medication on neural signatures:

  • Overall Cross-Site Accuracy: 69% ($P = 2.86 \times 10^{-13}$), which initially appeared promising.
  • Chronic Medicated Sensitivity: 71% ($P = 4.63 \times 10^{-5}$). The model successfully identified patients who matched the profile of the training data.
  • First-Episode Unmedicated Sensitivity: 48% ($P = 0.68$). This is essentially a coin flip. The model trained on chronic patients could not detect the disorder in patients who had not yet started medication.
  • First-Episode Medicated Sensitivity: 59% ($P = 0.10$). Performance improved slightly once medication was introduced, further supporting the hypothesis that the model was detecting medication effects.

Discussion: Medication vs. Disease Traits

Through univariable analysis, the authors found that medication usage had a significant impact on functional connectivity, whereas the duration of the illness did not. This is a "smoking gun" suggesting that medication is a massive confounding variable in psychiatric ML.

A true diagnostic tool needs to find "trait" biomarkers inherent to SSD biology. The authors argue that current models are instead picking up on the "medication state," making them less useful for early diagnosis—the very time when intervention is most effective. This "medication detector" effect poses a significant challenge for the clinical application of neuroimaging-based classifiers.

Limitations and Future Directions

While the study provides a critical reality check for the field, several limitations are noted:

  • Feature Diversity: The study focused solely on Functional Connectivity. Combining FC with structural MRI or diffusion tensor imaging (DTI) might yield more stable trait biomarkers.
  • Sample Size: While 44 unmedicated patients is a significant cohort for this field, larger samples are needed to train models specifically on early-stage pathology.
  • The Need for "Clean" Data: To find a true early-diagnosis biomarker, future ML models must be trained on large cohorts of unmedicated first-episode patients.

In conclusion, these findings suggest that we should reconsider current machine learning studies in chronic medicated SSD more cautiously. The path forward requires a shift in focus toward identifying the neural signatures of the disorder itself, independent of the transformative effects of long-term treatment.

Summary updated

Abstract

Abstract An accurate identification of schizophrenia spectrum disorder (SSD) at early stage could potentially allow for treating SSD with appropriate intervention to potentially prevent future deterioration. Despite mounting studies found neuroimaging combined with machine learning can identify chronic medicated SSD, whether or not the classification model identified the trait biomarker of SSD that can be used to identify early stage SSD is largely unknown. The present study aimed to investigate whether or not the classification model trained using chronic medicated SSD identified the trait biomarker of SSD that whether or not the model can be generalized to early stage SSD, by using functional connectivity (FC) combined with support vector machine (SVM) using a large sample from 4 independent sites (n = 1077). We found that the classification model trained using chronic medicated SSD from three sites(dataset 2, 3 and 4) classified SSD from HCs in another site (dataset 1) with 69% accuracy (P = 2.86e-13). Subgroup analysis indicated that this model can identify chronic medicated SSD in dataset 1 with 71% sensitivity (P = 4.63e-05), but cannot be generalized to first episode unmedicated SSD (sensitivity = 48%, P = 0.68) and first episode medicated SSD (sensitivity = 59%, P = 0.10). Univariable analysis showed that medication usage had significant effect on FC, but disease duration had no significant effect on FC. These findings suggest that the classification model trained using chronic medicated SSD may mainly identified the pattern of chronic medication usage state, rather than the trait biomarker of SSD. Therefore, we should reconsider the current machine learning studies in chronic medicated SSD more cautiously in term of the clinical application.
BibTeX
@misc{li2020classification,
  author = {Chao Li and Fei Wang and Xiaowei Jiang and Ji Chen and Jia Duan and Shaoqiang Han and Hao Yan and Yanqing Tang and Ke Xu},
  title = {{Classification of schizophrenia spectrum disorder using machine learning and functional connectivity: reconsidering the clinical application}},
  year = {2020},
  doi = {10.1101/2020.05.30.20118026},
  howpublished = {medRxiv},
}
Figure