Accuracy of diagnostic classification algorithms using cognitive-, electrophysiological-, and neuroanatomical data in antipsychotic-naïve schizophrenia patients

Bjørn H. Ebdrup, M. Axelsen, Nikolaj Bak, Birgitte Fagerlund, Bob Oranje, Jayachandra M. Raghava, Mette Ødegaard Nielsen, Egill Rostrup, Lars Kai Hansen, Birte GlenthøjPsychological 2018 · DOI

TL;DR

This study explored the diagnostic power of machine learning algorithms applied to multimodal data (cognition, electrophysiology, brain imaging) in distinguishing antipsychotic-naïve schizophrenia patients from healthy controls. Surprisingly, only cognitive data proved effective, suggesting its underappreciated role in schizophrenia diagnosis, while other modalities and their combinations failed to improve accuracy or predict symptom remission.

What Problem Does This Paper Address?

The current diagnostic framework for schizophrenia relies heavily on subjective clinical symptoms, which can be inconsistent and fail to capture the underlying biological complexity of the disorder. While numerous studies have identified objective biomarkers (cognitive deficits, electrophysiological abnormalities, structural brain changes) that differentiate schizophrenia patients from controls at a group level, these findings have not translated into clinical diagnostic tools. This gap highlights the need for advanced analytical methods, like machine learning, to integrate multimodal data and potentially create more objective and precise diagnostic approaches, especially in early stages of the illness before antipsychotic medication can confound results.

Methods

The researchers employed a sophisticated machine learning pipeline to analyze data from four modalities: neurocognitive tests, electrophysiology (EEG), structural MRI (sMRI), and diffusion tensor imaging (DTI). They utilized nine different configurations of supervised machine learning algorithms, including naive Bayes, logistic regression, support vector machines (SVM), decision trees, random forests, and auto-sklearn. A key aspect of their methodology was the use of a rigorous cross-validation strategy (100 stratified splits) to estimate generalization error and ensure robust performance evaluation. Missing data was handled through K-nearest neighbor imputation within the cross-validation loop. They first assessed the unimodal diagnostic accuracy of each modality and then explored multimodal approaches by combining cognition with other modalities. Finally, they investigated the predictive ability of these modalities for symptom remission after a 6-week course of amisulpride.

Datasets

The study utilized data from 46 antipsychotic-naïve, first-episode schizophrenia patients and 58 healthy controls. These participants underwent a comprehensive battery of assessments, including:

  • Cognition: 25 cognitive variables from tests like the Danish Adult Reading Test (DART), Wechsler Adult Intelligence Scale (WAIS III), Brief Assessment of Cognition in Schizophrenia (BACS), and Cambridge Neuropsychological Test Automated Battery (CANTAB).
  • Electrophysiology: 19 electrophysiological variables from the Copenhagen Psychophysiology Test Battery (CPTB), including P50 suppression, pre-pulse inhibition of the startle response (PPI), and mismatch negativity (MMN).
  • Neuroanatomy (sMRI): 98 structural MRI variables derived from T1-weighted images, focusing on regional gray matter volumes from an ROI approach.
  • Neuroanatomy (DTI): 100 diffusion tensor imaging variables, including fractional anisotropy (FA), mean diffusivity (MD), and other diffusion parameters extracted from white matter regions.

No new benchmark datasets were introduced; the study used existing data collected from participants.

Key Insights and Ideas

  • Cognition Reigns Supreme (for Diagnosis): The most striking finding is that only cognitive data significantly discriminated between schizophrenia patients and healthy controls. This challenges the common practice of excluding cognitive deficits from diagnostic criteria and suggests they are a core, early feature of the illness.
  • Multimodal Fusion Falls Short (in this context): Despite the promise of combining different data types, integrating cognitive data with electrophysiology, sMRI, or DTI did not improve diagnostic accuracy. This suggests that either the additional modalities don't carry independent diagnostic information in this early-stage, antipsychotic-naïve population, or the current analytical approaches aren't optimally leveraging them.
  • Early Stage is Key: The study's focus on antipsychotic-naïve, first-episode patients is crucial. The low accuracies for neuroimaging and electrophysiology might be due to the subtle nature of these changes at illness onset, contrasting with findings in more chronic or medicated populations where these markers might be more pronounced or confounded.
  • Symptom Remission Remains Elusive: None of the investigated modalities, including cognition and a constructed "clinical variable," could reliably predict short-term symptom remission. This points to the complexity of treatment response and the need for different predictive markers.

Key Results

  • Unimodal Diagnostic Accuracy:
    • Cognition: Achieved accuracies ranging from 60% to 69%, with all nine machine learning configurations significantly differentiating patients from controls (p values = 0.001-0.009).
  • Electrophysiology, sMRI, DTI: Accuracies ranged between 49% and 56%, failing to exceed chance level (56% for this dataset).
  • Multimodal Diagnostic Accuracy: Combinations of cognition with other modalities did not yield significantly higher accuracies than cognition alone, with accuracies ranging from 51% to 68%.
  • Prognostic Ability: None of the modalities predicted symptom remission after 6 weeks of amisulpride treatment, with accuracies ranging from 48% to 67%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Size: The relatively small sample size (46 patients) might have limited the power to detect subtle signals in electrophysiology and neuroimaging data, potentially contributing to the low accuracies.
  • Data Imputation: While imputation was performed carefully, it remains a potential source of bias.
  • Short-Term Remission: The prediction of symptom remission was based on a short-term outcome (6 weeks), and the criteria used were validated for longer-term remission.
  • Absence of fMRI and Neurochemistry: The study did not include functional MRI or neurochemical data, which might have provided additional insights.

Future work suggested includes:

  • Replication: The findings need to be replicated in larger, independent samples to confirm their robustness.
  • A Priori Variable Selection: For future machine learning applications, careful a priori variable selection based on independent data might be necessary to improve signal-to-noise ratio and enhance diagnostic accuracy.
  • Subgroup Identification: The authors encourage the use of multimodal, multivariate analyses to identify clinically meaningful subgroups of schizophrenia patients, potentially leading to more personalized treatment approaches.
  • Inclusion of Cognition in Diagnosis: The results strongly support the call for cognitive deficits to be considered in future diagnostic criteria for schizophrenia.

Summary updated

Abstract

In this multivariate and multimodal study in antipsychotic-naïve patients, only cognition significantly discriminated patients from controls, and no modality appeared to predict short-term symptom remission. Overall, these findings add to the increasing call for cognition to be included in the definition of schizophrenia. To bring about the full potential of machine learning algorithms in first-episode, antipsychotic-naïve schizophrenia patients, careful a priori variable selection based on independent data as well as inclusion of other modalities may be required.
BibTeX
@inproceedings{ebdrup2018accuracy,
  author = {Bjørn H. Ebdrup and M. Axelsen and Nikolaj Bak and Birgitte Fagerlund and Bob Oranje and Jayachandra M. Raghava and Mette Ødegaard Nielsen and Egill Rostrup and Lars Kai Hansen and Birte Glenthøj},
  title = {{Accuracy of diagnostic classification algorithms using cognitive-, electrophysiological-, and neuroanatomical data in antipsychotic-naïve schizophrenia patients}},
  booktitle = {Psychological},
  year = {2018},
  doi = {10.1017/s0033291718003781},
}
Figure