Classification with a Deferral Option and Low-Trust Filtering for Automated Seizure Detection

Thijs Becker, Kaat Vandecasteele, Christos Chatzichristos, Wim Van Paesschen, Dirk Valkenborg, Sabine Van Huffel, Maarten De VosSensors 2021 · DOI

TL;DR

This paper introduces two novel approaches to improve automated seizure detection in epilepsy using behind-the-ear EEG data. First, they propose a "deferral option" where uncertain predictions are sent to a human expert, achieving near-perfect detection sensitivity by deferring a portion of the data. Second, they enhance existing methods by filtering out "low-trust" predictions, significantly reducing false alarms while maintaining high detection rates.

What Problem Does This Paper Address?

Epilepsy affects a significant portion of the population, and accurately documenting seizures is crucial for patient care and treatment evaluation. While automated seizure detection systems hold great promise for long-term home monitoring, current methods struggle with achieving both high detection sensitivity (DS) and low false detection rates (FDR). Manual analysis of EEG data is time-consuming, and existing automated systems often have limitations in their accuracy, especially for individual patients. Furthermore, patient self-reporting of seizures is notoriously unreliable. This paper aims to bridge the gap between automated detection and clinical reliability by introducing methods that either leverage human expertise for difficult cases or intelligently filter out unreliable automated predictions.

Methods

The researchers propose two main strategies:

  • Classification with a Deferral Option: This approach leverages the idea that a human annotator is perfect. Instead of forcing the algorithm to make a decision on every segment, it identifies segments where its confidence is low. These segments are then "deferred" to a human expert for perfect classification. The confidence of the algorithm's prediction is determined using temperature scaling of the classification model's output or by calculating "trust scores." By deferring a certain percentage of the data, they aim to achieve near-perfect detection sensitivity.
  • Low-Trust Filtering (LTF): This method focuses on improving existing models that combine predictions from multiple short EEG segments. The core idea is that some of these short-term predictions might be "untrustworthy" due to noise or artifacts. LTF filters out these low-trust predictions before making a final decision for a longer segment. This is achieved by using trust scores, which are independent of the classifier itself and can identify atypical samples. By removing these unreliable predictions, the system aims to reduce false positives without significantly impacting detection sensitivity.

Datasets

The study utilizes a dataset of behind-the-ear EEG measurements from 54 epileptic patients with focal onset seizures, collected as part of the SeizeIT1 study. This dataset comprises approximately 220 days of EEG data and contains 114 seizures. The neurologist-annotated seizures from standard video-EEG recordings were used to label the data.

Key Insights and Ideas

  • Human-in-the-loop is powerful: Deferring even a moderate portion of the data (around 10-40%) to a human expert can lead to near-perfect seizure detection sensitivity. This highlights the value of combining automated analysis with human oversight for critical decisions.
  • Trust scores are more robust than raw confidences: For filtering out unreliable predictions (LTF), trust scores proved superior to the raw SVM confidences. This suggests that trust scores capture a more meaningful aspect of prediction uncertainty for this task.
  • Filtering low-trust predictions significantly reduces false alarms: LTF, particularly when using trust scores, led to a substantial decrease in false detection rates (21-43%) with only minor or no impact on detection sensitivity. This is a crucial finding for making automated systems more practical for real-world use.
  • LTF is a straightforward enhancement: The LTF approach requires optimizing only one hyper-parameter (the percentage of segments to filter), making it a relatively easy-to-implement post-processing step that can improve existing classifiers.

Key Results

  • Deferral Option:
    • A detection sensitivity of approximately 90% was achieved by deferring 10% of the data.
  • Deferring 40% of the data yielded a detection sensitivity of 99%.
  • Perfect detection sensitivity (100%) was achieved by deferring 50% of the data.
  • Low-Trust Filtering (LTF):
    • For the "clear ictal" SVM (CI SVM), LTF with trust scores reduced the FDR by 41% while slightly decreasing DS. PPV and F1-score saw significant improvements.
  • For the "full seizure" SVM (FS SVM), LTF with trust scores reduced FDR by 38% and slightly increased DS. PPV and F1-score also improved significantly.
  • LTF with SVM confidences did not improve performance and, in some cases, degraded the FDR.
  • The false detection rate showed a relative decrease between 21% and 43% with LTF.

Limitations and Future Work

  • Dataset Specificity: The study was conducted on a specific dataset of behind-the-ear EEG from patients with focal onset seizures. While the authors expect similar results for other seizure types, further validation on diverse datasets is needed.
  • Human Annotator Assumption: The deferral option assumes perfect human annotation, which is an idealization. In practice, human annotators can make errors.
  • Generalizing Deferral Strategies: The current deferral strategy involves selecting 5-minute segments. The authors suggest that more sophisticated deferral schemes, variable segment lengths, and more detailed property-based deferral choices could further improve results.
  • Better Confidence Measures: Developing improved confidence estimation methods is an area for future research.
  • Multi-modal Data: The authors suggest that incorporating multi-modal data (e.g., EMG, accelerometry) could enhance seizure detection, especially for certain seizure types.
  • Online Application: While LTF is computationally efficient, further research into real-time implementation and adaptation for online seizure detection is warranted.
  • Understanding Trust Score Drivers: The authors found it difficult to pinpoint the exact reasons for low trust scores in the EEG data, suggesting this is an area for further investigation.

Summary updated

BibTeX
@article{becker2021classification,
  author = {Thijs Becker and Kaat Vandecasteele and Christos Chatzichristos and Wim Van Paesschen and Dirk Valkenborg and Sabine Van Huffel and Maarten De Vos},
  title = {{Classification with a Deferral Option and Low-Trust Filtering for Automated Seizure Detection}},
  journal = {Sensors},
  year = {2021},
  doi = {10.3390/s21041046},
}
Figure