TL;DR
This paper investigates the reproducibility of schizophrenia (SZ) biomarkers identified by machine learning models trained on EEG data. By using explainable AI techniques on Random Forest (RF) and Support Vector Machine (SVM) models, the researchers found that different models highlight distinct EEG features, suggesting multiple pathways to SZ diagnosis and the potential for robust, cross-model biomarkers.
What Problem Does This Paper Address?
Schizophrenia (SZ) is a complex neuropsychiatric disorder affecting millions globally. Its diagnosis is challenging due to symptom variability and overlap with other conditions, making it reliant on subjective clinical assessment rather than objective biomarkers. This necessitates the development of automated diagnostic tools and the identification of reliable biomarkers to aid clinicians. While electroencephalography (EEG) is a promising modality due to its affordability and temporal resolution, existing studies often lack rigorous validation or clear explanations of their findings, hindering clinical translation. This paper aims to address this by exploring the reproducibility of SZ biomarkers identified by explainable machine learning models, seeking features that are consistently important across different modeling approaches.
Methods
The researchers employed a two-pronged approach: automated diagnosis and biomarker identification using explainable AI.
- Automated Diagnosis: They trained two machine learning models – a Random Forest (RF) and a Support Vector Machine (SVM) – to classify individuals with SZ from healthy controls (HCs) using spectral features extracted from resting-state EEG data.
- Biomarker Identification (Explainability): To understand why the models made their classifications, they used permutation feature importance. This involved systematically shuffling specific features (either frequency bands or electrode channels) and observing the impact on model performance. This allowed them to assess the relative importance of different EEG components.
- Spectral Importance: They analyzed the importance of different frequency bands (δ, θ, α, β, γlower, γupper).
- Spatial Importance: They analyzed the importance of individual EEG electrodes.
A key methodological aspect was the use of a robust, subject-based cross-validation strategy to ensure the generalizability of the models and prevent inflated performance metrics.
Datasets
The study utilized a publicly available scalp EEG dataset [14] comprising recordings from 14 individuals with SZ and 14 healthy controls (HCs). The data was collected at the Institute of Psychiatry and Neurology in Warsaw, Poland, with 64 electrodes placed in the standard 10-20 format during a 15-minute resting-state recording at a 250 Hz sampling rate. This dataset has been used in previous studies, providing a basis for comparison.
Key Insights and Ideas
- Multiple Pathways to Diagnosis: The study highlights that different machine learning models (RF vs. SVM) can identify distinct sets of EEG features as important for SZ diagnosis. This suggests that SZ might manifest in the brain's electrical activity in various ways, and no single feature set might capture all aspects.
- Reproducibility is Key for Biomarkers: By comparing the explanations from RF and SVM, the researchers aimed to find biomarkers that are consistently identified across models. This consistency is crucial for building confidence in their clinical utility.
- Spectral Features Hold Promise: The models achieved above-chance performance using spectral features, indicating that these frequency-based patterns in EEG are informative for SZ diagnosis.
- Robustness Varies: The RF model demonstrated greater robustness to the removal of individual electrode data compared to the SVM, suggesting it might be more resilient in real-world scenarios where data quality can vary.
Key Results
- Model Performance: Both the RF and SVM models performed significantly above chance level in classifying SZ. The RF generally outperformed the SVM across most metrics, particularly in specificity (correctly identifying HCs).
- RF: F1 score 72.54±09.85, Accuracy 74.99±08.19, Specificity 77.87±11.82
- SVM: F1 score 68.88±12.85, Accuracy 70.60±05.60, Specificity 59.99±19.88
- Spectral Biomarkers:
- The SVM prioritized the θ and α frequency bands.
- The RF heavily relied on the upper γ-band.
- These findings align with existing literature that points to differences in these bands between HCs and SZs.
- Spatial Biomarkers:
- The RF was more robust to the loss of individual channels.
- The SVM found the P4, T8, and P3 electrodes (temporal and parietal regions) to be most important for classification.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- Phase vs. Amplitude: The current analysis included both phase and amplitude information. Future work could specifically examine the relative importance of phase features versus amplitude features.
- Model Complexity: The study used two machine learning classifiers capable of identifying nonlinear patterns. It would be beneficial to compare their performance to simpler linear models, such as logistic regression.
- Dataset Generalizability: The reproducibility of findings across different datasets should be investigated to ensure the biomarkers are not dataset-specific.
- Feature Interactions: Exploring feature interactions using methods like SHAP could provide deeper insights into how different EEG components work together.
Summary updated