PEPbench—Open, Reproducible, and Systematic Benchmarking of Automated Pre‐Ejection Period Extraction Algorithms

Robert Richer, Julia Jorkowitz, Sebastian Stühler, Luca Abel, Miriam Kurz, Marie Oesten, Stefan G. Grießhammer, Nils C. Albrecht, Arne Küderle, Christoph Ostgathe, Alexander Koelpin, Tobias Steigleder, Nicolas Rohleder, Bjoern M. EskofierPsychophysiology 2025 · DOI

TL;DR

This paper introduces PEPbench, an open-source Python package and framework for systematically benchmarking automated pre-ejection period (PEP) extraction algorithms. By providing a standardized evaluation platform, a collection of algorithms, and two new publicly available annotated datasets, PEPbench aims to foster reproducibility and advance research in psychophysiology. The study reveals significant performance differences among algorithms, particularly highlighting the challenges and error sources in B-point detection.

What Problem Does This Paper Address?

The pre-ejection period (PEP) is a crucial physiological marker, often used to infer sympathetic nervous system (SNS) activity. It's measured as the time between ventricular depolarization (detected via ECG) and the opening of the aortic valve (detected via impedance cardiography, specifically the dZ/dt signal). While numerous algorithms exist for automatically extracting PEP, there's a significant lack of standardized benchmarking. This is largely due to the absence of open-source algorithms, reproducible evaluation frameworks, and publicly available, annotated datasets. This makes it difficult to compare algorithms objectively, hindering progress and reproducibility in psychophysiological research.

Methods

The core contribution is the PEPbench framework, a modular Python package built on tpcp. It offers:

  • Algorithm Integration: A standardized way to incorporate and run various Q-peak (ECG) and B-point (dZ/dt) detection algorithms, along with outlier correction methods.
  • Pipeline Construction: Tools to combine these algorithms into comprehensive PEP extraction pipelines.
  • Standardized Evaluation: A framework for running challenges (algorithm-dataset combinations) and computing a suite of evaluation metrics.
  • Visualization Tools: Functions for plotting data, algorithm outputs, and evaluation results.

The authors implemented 3 Q-peak algorithms and 12 B-point algorithms from existing literature, focusing on signal-processing-based approaches for fair comparison. They also included 2 outlier correction algorithms. A key methodological choice was annotating the Q-wave peak as the PEP start point, rather than the Q-wave onset, due to its greater reliability in noisy signals.

Datasets

The study introduces two new, publicly available datasets for benchmarking:

  • EmpkinS Dataset: Collected from 15 healthy participants during a modified Trier Social Stress Test (TSST) and a control condition (f-TSST), involving both standing and supine positions.
  • Guardian Dataset: Collected from 24 healthy participants during various autonomic challenges (resting, Valsalva maneuver, breath-holding, tilt-up, tilt-level) in a supine position.

Both datasets contain synchronized ECG and dZ/dt recordings with manually annotated Q-peaks and B-points, serving as ground truth. The authors also integrated two other existing datasets (ReBeatICG and TimeWindowICG) into PEPbench for broader compatibility, though they were not used for the core analysis due to annotation limitations.

Key Insights and Ideas

  • B-point Detection is the Bottleneck: The study strongly reinforces that B-point detection is the primary source of error and variability in automated PEP extraction. Algorithms for this step showed significantly higher error rates and inconsistency compared to Q-peak detection.
  • Algorithm Performance Varies Wildly: There isn't a single "best" algorithm across all scenarios. Performance depends heavily on the specific algorithm, dataset, and even individual participants.
  • Outlier Correction is Not a Universal Fix: While outlier correction methods can reduce invalid PEPs and improve robustness, their impact on accuracy (MAE) is inconsistent and can even worsen performance for some algorithms.
  • Simpler is Sometimes Better: The most complex algorithms weren't necessarily the best performers. Robustness to noise and artifacts in the dZ/dt signal appears more critical than algorithmic complexity.
  • Error Compensation in Pipelines: Combining Q-peak and B-point algorithms can sometimes lead to error cancellation, where positive and negative biases in individual steps offset each other, resulting in a lower overall error, even if the individual PEP values aren't perfectly accurate.
  • Open Science is Crucial: The paper champions open-source tools, reproducible frameworks, and public datasets as essential for advancing the field and fostering collaboration.

Key Results

  • Systematic Comparison: PEPbench facilitated the comparison of 108 different algorithm combinations.
  • Q-peak Algorithms: The Mar04 algorithm (Martinez et al., 2004) generally showed the lowest Mean Absolute Error (MAE) on both datasets (e.g., 5.3±9.5 ms on EmpkinS, 4.2±12.4 ms on Guardian). The Van13 algorithm (Van Lien et al., 2013) also performed well, particularly with adjusted time intervals. The For18 algorithm consistently underestimated PEP due to late Q-peak detection.
  • B-point Algorithms: The Dro22 algorithm (Drost et al., 2022) was the most robust, showing the best performance on EmpkinS (MAE 14.9±14.7 ms) and second-best on Guardian (MAE 18.2±15.2 ms). The Loz07LR algorithm (Lozano et al., 2007) performed best on Guardian (MAE 16.7±14.8 ms). Many algorithms produced a significant number of invalid PEPs (e.g., Pal21 failed in ~29-37% of cycles).
  • Combined Pipelines: The "Overall Lowest Error" pipeline on EmpkinS was For18 (Q-peak) | Dro22 (B-point) | None (Outlier Correction) with an MAE of 10.8±15.6 ms. On Guardian, it was For18 (Q-peak) | Dro22 (B-point) | None with an MAE of 11.6±15.9 ms. These pipelines outperformed "Individually Lowest Error" pipelines.
  • Inter-Rater Agreement: High agreement was observed between the two manual annotators for both Q-peaks and B-points, indicating reliable ground truth.

Limitations and Future Work

  • Dataset Limitations: The current datasets, while diverse, don't cover all possible experimental contexts (e.g., seated recordings, clinical populations). The authors encourage contributions of more diverse datasets.
  • Annotation Bias: The visual guidance during manual annotation (using Q-peak as a reference for B-point) might have introduced a subtle bias, though it aligns with established practices.
  • Algorithm Specificity: The effectiveness of outlier correction is highly algorithm-dependent and not universally beneficial.
  • C-point Detection: The reliance on automatic C-point detection (a precursor for many B-point algorithms) could introduce errors. Future work could explore manual C-point verification.
  • Sampling Rate: For some applications, higher sampling rates might be needed to reduce errors in B-point detection.
  • Future Work: The authors aim to expand PEPbench with more datasets and algorithms, and extend it to other cardiac parameters (e.g., LVET, TWA). They also emphasize the need for context-specific validation and further investigation into outlier correction's role. The ultimate goal is to foster a community-driven platform for robust, transparent, and collaborative research in psychophysiology.

Summary updated

Abstract

The pre-ejection period (PEP) is a widely used cardiac parameter in psychophysiology that reflects the duration between the onset of ventricular depolarization and the opening of the aortic valve. PEP is often used as a marker of cardiac sympathetic nervous system (SNS) activity, particularly in within-subject comparisons under similar hemodynamic conditions. While many algorithms for automated PEP extraction from electrocardiography (ECG) and impedance cardiography (ICG) signals (more precisely, its first derivative, dZ/dt) have been proposed in literature, they have not been systematically benchmarked. This lack of standardized algorithm comparisons originates from the absence of open-source algorithms and annotated datasets for evaluating PEP extraction algorithms. To address this issue, we introduce PEPbench, an open-source Python package with different Q-peak and B-point detection algorithms from literature that can be combined to create comprehensive PEP extraction pipelines, and a standardized framework for evaluating PEP extraction algorithms. We use PEPbench to systematically compare 108 different algorithm combinations. All combinations are evaluated on two datasets with manually annotated Q-peaks and B-points, which we make publicly available as the first datasets with reference PEP annotations. Our results show that the algorithms can differ vastly in their performance and that B-point detection algorithms introduce a considerable amount of error. Thus, we suggest that automated PEP extraction algorithms should be used with caution on a beat-to-beat level as their error rates are relatively high. This highlights the need for open and reproducible benchmarking frameworks for PEP extraction algorithms to improve the quality of research findings in the field of psychophysiology. With PEPbench, we aim to take a first step toward this goal and encourage other researchers to engage in the evaluation of PEP extraction algorithms by contributing algorithms, data, and annotations. We hope to establish a community-driven platform, fostering innovation and collaboration in the field of psychophysiology and beyond.
BibTeX
@inproceedings{richer2025pepbenchopen,
  author = {Robert Richer and Julia Jorkowitz and Sebastian Stühler and Luca Abel and Miriam Kurz and Marie Oesten and Stefan G. Grießhammer and Nils C. Albrecht and Arne Küderle and Christoph Ostgathe and Alexander Koelpin and Tobias Steigleder and Nicolas Rohleder and Bjoern M. Eskofier},
  title = {{PEPbench—Open, Reproducible, and Systematic Benchmarking of Automated Pre‐Ejection Period Extraction Algorithms}},
  booktitle = {Psychophysiology},
  year = {2025},
  doi = {10.1111/psyp.70176},
}
Figure