Weak supervision as an efficient approach for automated seizure detection in electroencephalography

Khaled Saab, Jared Dunnmon, Christopher Ré, Daniel L. Rubin, Christopher Lee-Messer

TL;DR

This paper introduces a novel approach to automated seizure detection in EEG by leveraging "weak" annotations – imperfect but readily available labels from clinical workflows – to train deep learning models. This method achieves high performance comparable to or exceeding current clinical software, offering a sustainable and scalable solution for developing seizure detection systems without the need for expensive, manually curated datasets.

What Problem Does This Paper Address?

Epileptic seizures are a significant cause of morbidity worldwide, and accurate detection through electroencephalography (EEG) is crucial for patient care and treatment decisions. However, manually analyzing EEG data is incredibly time-consuming and costly, leading to a shortage of interpreters and limiting the widespread use of continuous EEG monitoring. While deep learning models show promise, they typically require vast amounts of precisely labeled data, which are difficult and expensive to obtain in the medical domain. This paper tackles the challenge of creating effective automated seizure detection models by exploring the use of "weak supervision," a method that utilizes imperfect, but abundant, annotations already generated during routine clinical practice.

Methods

The core of this work lies in using "weak annotations" to train Convolutional Neural Networks (CNNs) for automated, real-time EEG seizure detection. These weak annotations are preliminary labels provided by a mixed group of clinical staff (technicians, fellows, students, and epileptologists) as part of their daily workflow. Unlike "gold-standard" labels created by expert epileptologists, weak annotations are less precise (with reported precision scores as low as 0.37) and may not capture all seizures.

The researchers trained CNN models on these weak annotations, treating the problem as a classification task for 12-second (fast detection) and 60-second (slow detection) EEG clips. They employed a densely connected inception architecture, inspired by existing deep learning models, to process the EEG signals. The training process involved balancing the dataset by undersampling negative examples (non-seizure clips) to address the inherent class imbalance.

Datasets

The study utilized two large, real-world EEG datasets:

  • Stanford Clinical Datasets:
    • Pediatric: 5,076 unique pediatric patients with 36,644 EEG signals.
  • Adult: 7,363 unique adult patients with 99,721 EEG signals. These datasets contain "weak annotations" collected during routine clinical workflows. For evaluation, a subset of these signals was annotated by a fellowship-trained EEG reader (CLM) to serve as gold-standard labels.

  • Temple University Hospital (TUH) EEG Dataset: A publicly available dataset used for evaluating model generalization across institutions. This dataset contains approximately 2,012 seizures.

Key Insights and Ideas

  • Weak Supervision is Powerful: The paper demonstrates that imperfect, but readily available, "weak" annotations can be sufficient to train high-performing seizure detection models, circumventing the need for expensive expert labeling.
  • Clinical Workflow Integration: By leveraging annotations already generated in the clinical workflow, the proposed method offers a sustainable and scalable pipeline for continuous model improvement without additional resource allocation.
  • Generalization Challenges: Models trained on one institution's data do not always generalize well to another's, highlighting the need for institution-specific or adaptable models.
  • Slow Detection Advantage: For adult patients, the 60-second clip detection approach significantly outperformed the 12-second approach, suggesting that a broader temporal context is beneficial for accurate detection in this population.

Key Results

  • High Performance: The CNN models trained with weak supervision achieved AUROC values of 0.93 for pediatric and 0.94 for adult seizure onset detection (using 60-second clips).
  • Outperforming Baselines: The weakly supervised CNN model substantially outperformed traditional machine learning classifiers (Logistic Regression, Random Forest) and a leading commercial software (Persyst-13) in terms of F1-score. Specifically, it showed an 18-point increase in F1-score for pediatric patients and an 11-point increase for adult patients compared to Persyst-13.
  • Scalability: The study showed that model performance improves with the amount of weakly labeled training data, demonstrating the benefits of larger datasets.
  • Generalization: Models showed a notable decrease in performance when tested on data from a different age group (pediatric vs. adult) or a different institution, underscoring the importance of training data diversity and potential for transfer learning.

Limitations and Future Work

  • Generalization Across Institutions: The paper acknowledges that models trained on data from a single institution may not perform as well on data from unseen institutions due to variations in sensing equipment, protocols, and patient demographics. Future work could involve transfer learning or multitask learning to improve cross-institutional generalization.
  • Age-Specific Performance: There was a significant performance drop when models trained on one age group were tested on the other, suggesting that age-naïve models may not reliably generalize.
  • Seizure Subtype Detection: The current work focuses on seizure onset detection. Future research aims to extend this to classifying seizure subtypes.
  • Temporal Information: The clip-based approach has limitations in resolving ambiguities in early seizure patterns when information from later time points is crucial. Future work could explore models that better integrate longer temporal contexts.
  • Data Availability: The Stanford clinical datasets are not publicly available due to Protected Health Information restrictions.

Summary updated

Abstract

Automated seizure detection from electroencephalography (EEG) would improve the quality of patient care while reducing medical costs, but achieving reliably high performance across patients has proven difficult. Convolutional Neural Networks (CNNs) show promise in addressing this problem, but they are limited by a lack of large labeled training datasets. We propose using imperfect but plentiful archived annotations to train CNNs for automated, real-time EEG seizure detection across patients. While these weak annotations indicate possible seizures with precision scores as low as 0.37, they are commonly produced in large volumes within existing clinical workflows by a mixed group of technicians, fellows, students, and board-certified epileptologists. We find that CNNs trained using such weak annotations achieve Area Under the Receiver Operating Characteristic curve (AUROC) values of 0.93 and 0.94 for pediatric and adult seizure onset detection, respectively. Compared to currently deployed clinical software, our model provides a 31% increase (18 points) in F1-score for pediatric patients and a 17% increase (11 points) for adult patients. These results demonstrate that weak annotations, which are sustainably collected via existing clinical workflows, can be leveraged to produce clinically useful seizure detection models.
BibTeX
@article{saab2020weak,
  author = {Khaled Saab and Jared Dunnmon and Christopher Ré and Daniel L. Rubin and Christopher Lee-Messer},
  title = {{Weak supervision as an efficient approach for automated seizure detection in electroencephalography}},
  journal = {Npj Digit. Medicine},
  year = {2020},
  doi = {10.1038/s41746-020-0264-0},
}
Figure