Generalization Error of a Regression Model for Non-Invasive Blood Pressure Monitoring using a Single Photoplethysmography (PPG) Signal

Marek Żyliński, Edoardo Occhipinti, Danilo P. Mandic2023 · DOI

TL;DR

This paper exposes a critical "generalization gap" in wearable blood pressure monitoring: models that appear highly accurate in standard cross-validation often fail completely when tested on new subjects or different time periods. The researchers demonstrate that while a PPG-based model can "learn" a specific person's blood pressure patterns, it struggles to generalize, often reverting to predicting a simple average value when faced with truly unseen data.

What Problem Does This Paper Address?

The quest for "cuffless" blood pressure (BP) monitoring is the holy grail of wearable health tech. Currently, most research uses Photoplethysmography (PPG)—the same green-light technology in your smartwatch—to estimate BP. While many papers report impressive accuracy, there is a massive disconnect between lab results and real-world reliability.

The authors argue that the common practice of cross-validation (where data from the same subjects are shuffled into both training and testing sets) creates a mirage of high performance. Because the model has already "seen" the physiological signature of a subject during training, it isn't actually learning to predict BP from PPG features; it’s simply memorizing that specific individual's baseline. This paper investigates how performance collapses when the model is forced to generalize to new data.

Methods

The researchers built a regression model using ensemble trees to predict Diastolic Blood Pressure (DBP).

  1. Feature Extraction: They extracted 16 distinct features from the PPG signal and its second derivative. These included morphological markers like peak-to-peak amplitude, "areas under the curve" (S1-S4) representing different phases of the heartbeat, and timing intervals of the "A" and "B" waves which relate to arterial stiffness and pressure wave propagation.
  2. Validation Strategies: To test generalization, they compared two data-splitting methods:
  3. Random Split: A standard 4-fold cross-validation where data from all subjects are shuffled. The model effectively "knows" every subject in the test set.
  4. Sequential Split: The dataset is divided into four chronological parts. This tests the model on a mix of seen and unseen subjects or different physiological states (like post-exercise vs. rest), providing a much harsher, more realistic test of external validation.

Datasets

The study utilized the "Continuous Cuffless Monitoring of Arterial Blood Pressure via Graphene Bioimpedance Tattoos" dataset, available via PhysioNet.

  • Subjects: 7 individuals.
  • Conditions: Data was recorded during various maneuvers designed to fluctuate blood pressure, including handgrips, cold pressor tests, Valsalva maneuvers, and cycling.
  • Ground Truth: Continuous BP was measured using a professional-grade finger cuff (Finapres NOVA) for comparison.

Key Insights and Ideas

  • The "Average Value" Trap: In external validation, the model essentially gives up on "tracking" the BP signal. Instead of following the ups and downs of the actual blood pressure, the predicted values cluster around a single average. This minimizes the mathematical error (MAE) but renders the device useless for actual monitoring.
  • Linear Dependency vs. Randomness: In random cross-validation, predicted and real values show a clear linear correlation. In external validation, this relationship vanishes. This suggests the model is over-fitting to individual-specific PPG signatures rather than universal physiological laws of hemodynamics.
  • PPG is Not Enough: The authors hypothesize that the PPG signal alone might lack the "necessary information" to estimate DBP across a diverse population without individual calibration. Factors like skin tone, sensor mounting pressure, and age-related arterial compliance vary too much between people for a "one-size-fits-all" model.

Key Results

  • Random Split (The "Optimistic" View): The model achieved a consistent Mean Absolute Error (MAE) of 7.4–7.7 mmHg. This is often considered "good" in literature.
  • Sequential Split (The "Realistic" View): Performance was highly volatile. While some segments showed low error (4.3 mmHg), others spiked to an unusable 35.2 mmHg.
  • Visual Evidence: Scatter plots (Figure 2) clearly show that while the random split results follow a 45-degree line (perfect prediction), the sequential split results look like a horizontal cloud—indicating the model is just guessing the mean.

Limitations and Future Work

The primary limitation acknowledged is the small dataset size (only 7 subjects). Machine learning models generally require thousands of diverse subjects to find universal patterns.

Future Directions: * Individual Calibration: The authors suggest that models might need a "calibration phase" for each user to account for their unique arterial properties. * Multi-Modal Sensing: Combining PPG with Electrocardiogram (ECG) data might provide the missing information (like Pulse Arrival Time) needed for more robust BP estimation. * Better Pre-processing: More sophisticated algorithms are needed to isolate "artery-only" information from the PPG signal, filtering out noise from skin deformation and sensor movement.

Summary updated

Abstract

Photoplethysmography (PPG) sensors integrated in wearable devices offer the potential to monitor arterial blood pressure (ABP) in patients. Such cuffless, non-invasive, and continuous solution is suitable for remote and ambulatory monitoring. A machine learning model based on PPG signal can be used to detect hypertension, estimate beat-by-beat ABP values, and even reconstruct the shape of the ABP. Overall, models presented in literature have shown good performance, but there is a gap between research and potential real-world use cases. Usually, models are trained and tested on data from the same dataset and same subjects, which may lead to overestimating their accuracy. In this paper: we compare cross-validation, where the test data are from the same dataset as training data, and external validation, where the model is tested on samples from a new dataset, on a regression model which predicts diastolic blood pressure from PPG features. The results show that, in the cross-validation, the predicted and the real values are linearly dependent, while in the external validation, the predicted values are not related to the real ones, but probably just through an average value.
BibTeX
@misc{zylinski2023generalization,
  author = {Marek Żyliński and Edoardo Occhipinti and Danilo P. Mandic},
  title = {{Generalization Error of a Regression Model for Non-Invasive Blood Pressure Monitoring using a Single Photoplethysmography (PPG) Signal}},
  year = {2023},
  doi = {10.1109/embc40787.2023.10340929},
}
Figure