Examining Reproducibility of EEG Schizophrenia Biomarkers Across Explainable Machine Learning Models

Charles A. Ellis, Abhinav Sattiraju, Robyn L. Miller, Vince D. Calhoun

TL;DR

This paper investigates the reproducibility of schizophrenia (SZ) biomarkers identified by machine learning models trained on EEG data. By using explainable AI techniques on Random Forest (RF) and Support Vector Machine (SVM) models, the researchers found that different models highlight distinct EEG features, suggesting multiple pathways to SZ diagnosis and the potential for robust, cross-model biomarkers.

What Problem Does This Paper Address?

Schizophrenia (SZ) is a complex neuropsychiatric disorder affecting millions globally. Its diagnosis is challenging due to symptom variability and overlap with other conditions, making it reliant on subjective clinical assessment rather than objective biomarkers. This necessitates the development of automated diagnostic tools and the identification of reliable biomarkers to aid clinicians. While electroencephalography (EEG) is a promising modality due to its affordability and temporal resolution, existing studies often lack rigorous validation or clear explanations of their findings, hindering clinical translation. This paper aims to address this by exploring the reproducibility of SZ biomarkers identified by explainable machine learning models, seeking features that are consistently important across different modeling approaches.

Methods

The researchers employed a two-pronged approach: automated diagnosis and biomarker identification using explainable AI.

  1. Automated Diagnosis: They trained two machine learning models – a Random Forest (RF) and a Support Vector Machine (SVM) – to classify individuals with SZ from healthy controls (HCs) using spectral features extracted from resting-state EEG data.
  2. Biomarker Identification (Explainability): To understand why the models made their classifications, they used permutation feature importance. This involved systematically shuffling specific features (either frequency bands or electrode channels) and observing the impact on model performance. This allowed them to assess the relative importance of different EEG components.
  3. Spectral Importance: They analyzed the importance of different frequency bands (δ, θ, α, β, γlower, γupper).
  4. Spatial Importance: They analyzed the importance of individual EEG electrodes.

A key methodological aspect was the use of a robust, subject-based cross-validation strategy to ensure the generalizability of the models and prevent inflated performance metrics.

Datasets

The study utilized a publicly available scalp EEG dataset [14] comprising recordings from 14 individuals with SZ and 14 healthy controls (HCs). The data was collected at the Institute of Psychiatry and Neurology in Warsaw, Poland, with 64 electrodes placed in the standard 10-20 format during a 15-minute resting-state recording at a 250 Hz sampling rate. This dataset has been used in previous studies, providing a basis for comparison.

Key Insights and Ideas

  • Multiple Pathways to Diagnosis: The study highlights that different machine learning models (RF vs. SVM) can identify distinct sets of EEG features as important for SZ diagnosis. This suggests that SZ might manifest in the brain's electrical activity in various ways, and no single feature set might capture all aspects.
  • Reproducibility is Key for Biomarkers: By comparing the explanations from RF and SVM, the researchers aimed to find biomarkers that are consistently identified across models. This consistency is crucial for building confidence in their clinical utility.
  • Spectral Features Hold Promise: The models achieved above-chance performance using spectral features, indicating that these frequency-based patterns in EEG are informative for SZ diagnosis.
  • Robustness Varies: The RF model demonstrated greater robustness to the removal of individual electrode data compared to the SVM, suggesting it might be more resilient in real-world scenarios where data quality can vary.

Key Results

  • Model Performance: Both the RF and SVM models performed significantly above chance level in classifying SZ. The RF generally outperformed the SVM across most metrics, particularly in specificity (correctly identifying HCs).
  • RF: F1 score 72.54±09.85, Accuracy 74.99±08.19, Specificity 77.87±11.82
    • SVM: F1 score 68.88±12.85, Accuracy 70.60±05.60, Specificity 59.99±19.88
  • Spectral Biomarkers:
    • The SVM prioritized the θ and α frequency bands.
  • The RF heavily relied on the upper γ-band.
  • These findings align with existing literature that points to differences in these bands between HCs and SZs.
  • Spatial Biomarkers:
    • The RF was more robust to the loss of individual channels.
  • The SVM found the P4, T8, and P3 electrodes (temporal and parietal regions) to be most important for classification.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Phase vs. Amplitude: The current analysis included both phase and amplitude information. Future work could specifically examine the relative importance of phase features versus amplitude features.
  • Model Complexity: The study used two machine learning classifiers capable of identifying nonlinear patterns. It would be beneficial to compare their performance to simpler linear models, such as logistic regression.
  • Dataset Generalizability: The reproducibility of findings across different datasets should be investigated to ensure the biomarkers are not dataset-specific.
  • Feature Interactions: Exploring feature interactions using methods like SHAP could provide deeper insights into how different EEG components work together.

Summary updated

Abstract

Abstract Schizophrenia (SZ) is a neuropsychiatric disorder that adversely effects millions of individuals globally. Current diagnostic efforts are symptom based and hampered due to the variability in symptom presentation across individuals and overlap of symptoms with other neuropsychiatric disorders. This spawns the need for (1) biomarkers to aid with empirical SZ diagnosis and (2) the development of automated diagnostic approaches that will eventually serve in a clinical decision support role. In this study, we train random forest (RF) and support vector machine (SVM) models to differentiate between individuals with schizophrenia and healthy controls using spectral features extracted from resting state EEG data. We then perform two explainability analyses to gain insight into key frequency bands and channels. In our explainability analyses, we examine the reproducibility of SZ biomarkers across models with the goal of identifying those that have potential clinical implications. Our model performance results are well above chance level indicating the broader utility of spectral information for SZ diagnosis. Additionally, we find that the RF prioritizes the upper γ-band and is robust to loss of information from individual electrodes, while the SVM prioritizes the α and θ-bands and P4 and T8 electrodes. It is our hope that our findings will inform future efforts towards the empirical diagnosis of SZ and towards the development of clinical decision support systems for SZ diagnosis.
BibTeX
@inproceedings{ellis2022examining,
  author = {Charles A. Ellis and Abhinav Sattiraju and Robyn L. Miller and Vince D. Calhoun},
  title = {{Examining Reproducibility of EEG Schizophrenia Biomarkers Across Explainable Machine Learning Models}},
  booktitle = {BIBE},
  year = {2022},
  doi = {10.1109/bibe55377.2022.00069},
}
Figure