ALPEC: A Comprehensive Evaluation Framework and Dataset for Machine Learning-Based Arousal Detection in Clinical Practice

Stefan Kraft, Andreas Theissler, Vera Wienhausen-Wilke, Philipp Walter, Gjergji KasneciarXiv 2024 · DOI · arXiv

Detecting arousals—brief shifts to lighter sleep or wakefulness—is a cornerstone of diagnosing sleep disorders like obstructive sleep apnea. However, a significant "translation gap" exists between how clinicians identify these events and how machine learning (ML) researchers build models to detect them.

The paper "ALPEC: A Comprehensive Evaluation Framework and Dataset for Machine Learning-Based Arousal Detection in Clinical Practice" aims to close this gap by rethinking both the task and how we measure success.

The Problem: Misalignment in Arousal Detection Research

In a clinical sleep lab, experts typically annotate only the onset (the start) of an arousal. Most existing ML research, however, treats arousal detection as a segmentation task, requiring precise "start" and "stop" times. This forces researchers to use datasets with "synthetic" durations (often just a default 3 seconds), creating a mismatch between training data and clinical reality.

Furthermore, standard ML metrics like pointwise accuracy or Area Under the Precision-Recall Curve (AUPRC) do not translate well to clinical utility. A physician using a decision-support tool prioritizes two things:

  1. Did the model find the correct number of arousals?
  2. Are the detections in the right neighborhood to allow for rapid verification?

Current evaluation methods are fragmented, often failing to account for class imbalance or the specific temporal constraints of sleep medicine.

Table 1: Comparison of this work against the fragmented landscape of previous arousal detection research.

Methodology: The ALPEC Framework

The authors propose a two-pronged solution to align ML with clinical needs:

1. Interval-based Onset Detection (IOD)

Instead of predicting a precise duration, models are trained to detect a 10-second interval centered around the clinical onset. The researchers utilize the DeepSleep architecture—a Fully Convolutional Network based on U-Net—for continuous segmentation of sleep signals.

2. The ALPEC Evaluation Pipeline

ALPEC stands for Approximate Localization and Precise Event Count. This framework includes:

  • Post-processing: Merging predicted intervals that are too close together (as arousals must be separated by 10s of stable sleep) and discarding unrealistically long intervals (>60s).
  • Evaluation: Using a 15-second "temporal tolerance buffer" around ground-truth onsets. If a predicted interval overlaps this buffer, it is counted as a True Positive. The F2-score is the primary metric, weighting recall higher than precision to reflect the clinical preference for high sensitivity over missing critical events.

The Comprehensive Polysomnography (CPS) Dataset

The study introduces the CPS dataset, a rich resource comprising 113 diagnostic recordings from a German sleep laboratory. Unlike most public datasets, CPS includes:

  • Up to 36 raw and 23 derived channels.
  • Unique Modalities: Beat-by-beat blood pressure estimations and pulse transit time, allowing for the exploration of cardiovascular signals as indicators of arousal.

Key Results and Insights

The results demonstrate that traditional "window-based evaluation" (WE) significantly overestimates model performance.

Table 2: Comparative evaluation of different models. Note how ALPEC provides a more conservative and realistic performance estimate than Window-based Evaluation (WE).

Key findings include:

  • Honest Assessment: A "Constant 1" baseline (predicting an arousal in every window) achieves a deceptive F2 of 0.53 under standard WE but correctly drops to 0.00 under ALPEC.
  • Top Performance: The best model (D4), which leveraged the full suite of respiratory and cardiovascular signals, achieved an F2-score of 0.73 on the CPS dataset using ALPEC evaluation.
  • Multimodal Value: Moving from a single-channel EEG model (D1, F2=0.65) to the multimodal D4 model (F2=0.73) showed a clear performance gain, validating the inclusion of additional sensors in the CPS dataset.

Conclusion and Future Directions

By shifting the focus from duration to onsets and providing a more stringent evaluation framework, ALPEC offers a path toward integrating ML into clinical workflows. Future work will focus on:

  • Classification of Causes: Identifying whether an arousal is respiratory or spontaneous.
  • Explainability: Highlighting which signals triggered a detection to aid clinical decision-making.
  • Standardization: Validating parameters like the 15s tolerance buffer with a broader group of sleep experts.

This research highlights that for ML to be useful in medicine, it must be evaluated through the lens of clinical utility rather than just statistical accuracy.

Summary updated

Abstract

Detecting arousals in sleep is essential for diagnosing sleep disorders. However, using Machine Learning (ML) in clinical practice is impeded by fundamental issues, primarily due to mismatches between clinical protocols and ML methods. Clinicians typically annotate only the onset of arousals, while ML methods rely on annotations for both the beginning and end. Additionally, there is no standardized evaluation methodology tailored to clinical needs for arousal detection models. This work addresses these issues by introducing a novel post-processing and evaluation framework emphasizing approximate localization and precise event count (ALPEC) of arousals. We recommend that ML practitioners focus on detecting arousal onsets, aligning with clinical practice. We examine the impact of this shift on current training and evaluation schemes, addressing simplifications and challenges. We utilize a novel comprehensive polysomnographic dataset (CPS) that reflects the aforementioned clinical annotation constraints and includes modalities not present in existing polysomnographic datasets. We release the dataset alongside this paper, demonstrating the benefits of leveraging multimodal data for arousal onset detection. Our findings significantly contribute to integrating ML-based arousal detection in clinical settings, reducing the gap between technological advancements and clinical needs.
BibTeX
@misc{kraft2024alpec,
  author = {Stefan Kraft and Andreas Theissler and Vera Wienhausen-Wilke and Philipp Walter and Gjergji Kasneci},
  title = {{ALPEC: A Comprehensive Evaluation Framework and Dataset for Machine Learning-Based Arousal Detection in Clinical Practice}},
  year = {2024},
  eprint = {2409.13367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.13367},
}
Figure