Machine-Learning-Based Diagnostics of EEG Pathology

Lukas Alexander Wilhelm Gemein, Robin Tibor Schirrmeister, Patryk Chrabaszcz, Daniel Wilson, Joschka Boedecker, Andreas Schulze-Bonhage, Frank Hutter, Tonio Ball

TL;DR

This paper presents a comprehensive comparison of feature-based and end-to-end deep learning methods for diagnosing EEG pathology. The study finds that both approaches achieve similar, high accuracies (81-86%), suggesting that current methods may be approaching a performance ceiling due to inherent label noise in clinical EEG data.

What Problem Does This Paper Address?

Electroencephalogram (EEG) analysis is crucial for diagnosing neurological conditions, but it's a time-consuming and expertise-intensive process. Machine learning (ML) offers a promising avenue for automating this analysis, but the field has been divided between "feature-based" methods (relying on handcrafted features) and "end-to-end" deep learning approaches (learning features directly from raw data). Previous comparisons have often been limited, leading to an overemphasis on deep learning's superiority. This paper aims to provide a robust, systematic comparison of these two paradigms on a large, publicly available EEG dataset, addressing the question of which approach is more effective and whether deep learning truly offers a significant advantage.

Methods

The authors developed a comprehensive feature-based framework that extracts a wide array of features across time, frequency, and connectivity domains. They then applied several classifiers to these features, including Random Forest (RF), Support Vector Machine (SVM), and an automated ML toolkit (auto-sklearn). For the end-to-end approach, they utilized established deep neural network architectures: a four-layered Convolutional Neural Network (CNN) called BD-Deep4, an EEG-optimized Temporal Convolutional Network (TCN) called BD-TCN, and two other CNNs (BD-Shallow and BD-EEGNet). A key aspect of their methodology was the rigorous comparison of these diverse methods on the same dataset, employing cross-validation and a held-out final evaluation set. They also conducted analyses to understand feature importance and network behavior.

Datasets

The primary dataset used in this study is the Temple University Hospital (TUH) Abnormal EEG Corpus (v2.0.0). This corpus contains approximately 3000 EEG recordings of at least 15 minutes duration from 2329 unique patients. It includes both pathological and non-pathological recordings, with physician reports providing additional diagnostic information. The dataset is publicly available for download.

Key Insights and Ideas

  • Performance Saturation: The most striking insight is that both sophisticated feature-based methods (especially Riemannian geometry) and advanced deep learning models achieve remarkably similar, high accuracies (81-86%). This suggests that current ML approaches for EEG pathology decoding might be approaching a performance limit.
  • The Ceiling of Label Noise: The authors hypothesize that this performance ceiling is largely due to inherent label noise in clinical EEG data. Inter-rater agreement among neurologists is only moderate, meaning the "ground truth" labels themselves are imperfect, limiting the achievable accuracy of any classifier.
  • Feature-Based Methods Still Relevant: Contrary to some literature, this study demonstrates that well-engineered feature-based approaches can be highly competitive with deep learning, especially when a broad set of informative features is utilized.
  • Temporal and Delta/Theta Bands are Key: Both feature importance analysis and network perturbation studies highlight the importance of temporal electrode locations and low-frequency bands (delta and theta) for pathology detection.
  • Interpretability vs. Performance Trade-off: While deep learning models are often seen as "black boxes," the study found that even with extensive feature engineering, understanding why a model makes a prediction can be challenging due to high feature correlations.

Key Results

  • Overall Accuracy: The best feature-based decoding accuracy was achieved by Riemannian-geometry-based decoding at 85.87%. The best end-to-end deep learning model, BD-TCN, achieved 86.16% accuracy.
  • Narrow Accuracy Range: Across all tested models (feature-based and deep learning), accuracies fell within a narrow range of 81-86%.
  • No Statistical Difference: Statistical analysis revealed no significant difference in performance between the various models investigated.
  • False Negatives: All models tended to produce more false negatives (classifying pathological cases as normal) than false positives.
  • Feature Importance: Temporal electrodes (T3, T4) and low-frequency bands (delta, theta) were found to be highly informative for pathology detection.

Limitations and Future Work

  • Label Noise: The primary limitation acknowledged is the inherent label noise in the dataset, which likely caps achievable accuracy.
  • Beyond Binary Classification: The current binary classification (pathological vs. non-pathological) is a simplification. The authors suggest that decoding individual pathologies or using more nuanced labels (e.g., normal, epileptic, other pathological) could be more clinically useful.
  • Incorporating Clinical Information: Clinicians use more than just EEG data (e.g., patient history, age, gender, imaging). The authors suggest that integrating such information could improve diagnostic accuracy, though their initial attempts to include age and gender yielded only marginal improvements.
  • Data Augmentation: Exploring data augmentation techniques for EEG could potentially improve model generalization, especially for end-to-end methods which are data-intensive.
  • Reproducibility: The authors emphasize the importance of public availability of code and data for scientific rigor and have made their code and data processing pipelines available.

Summary updated

Abstract

Machine learning (ML) methods have the potential to automate clinical EEG analysis. They can be categorized into feature-based (with handcrafted features), and end-to-end approaches (with learned features). Previous studies on EEG pathology decoding have typically analyzed a limited number of features, decoders, or both. For a I) more elaborate feature-based EEG analysis, and II) in-depth comparisons of both approaches, here we first develop a comprehensive feature-based framework, and then compare this framework to state-of-the-art end-to-end methods. To this aim, we apply the proposed feature-based framework and deep neural networks including an EEG-optimized temporal convolutional network (TCN) to the task of pathological versus non-pathological EEG classification. For a robust comparison, we chose the Temple University Hospital (TUH) Abnormal EEG Corpus (v2.0.0), which contains approximately 3000 EEG recordings. The results demonstrate that the proposed feature-based decoding framework can achieve accuracies on the same level as state-of-the-art deep neural networks. We find accuracies across both approaches in an astonishingly narrow range from 81--86\%. Moreover, visualizations and analyses indicated that both approaches used similar aspects of the data, e.g., delta and theta band power at temporal electrode locations. We argue that the accuracies of current binary EEG pathology decoders could saturate near 90\% due to the imperfect inter-rater agreement of the clinical labels, and that such decoders are already clinically useful, such as in areas where clinical EEG experts are rare. We make the proposed feature-based framework available open source and thus offer a new tool for EEG machine learning research.
BibTeX
@article{gemein2020machine,
  author = {Lukas Alexander Wilhelm Gemein and Robin Tibor Schirrmeister and Patryk Chrabaszcz and Daniel Wilson and Joschka Boedecker and Andreas Schulze-Bonhage and Frank Hutter and Tonio Ball},
  title = {{Machine-Learning-Based Diagnostics of EEG Pathology}},
  journal = {NeuroImage},
  year = {2020},
  doi = {10.1016/j.neuroimage.2020.117021},
}
Figure