The reliability of remote photoplethysmography under low illumination and elevated heart rates

Bhargav Acharya, William Saakyan, Barbara Hammer, Hanna DrimallaNpj Digit. Medicine 2025 · DOI

TL;DR

This paper introduces the CHILL dataset, a novel benchmark for evaluating remote photoplethysmography (rPPG) under challenging low-illumination and elevated heart rate conditions. Their findings reveal significant performance drops in current rPPG methods under these realistic scenarios, highlighting a critical need for more robust algorithms for digital health applications.

What Problem Does This Paper Address?

Remote photoplethysmography (rPPG) offers a promising non-invasive way to measure heart rate using standard cameras, making it ideal for telemedicine and digital health. However, existing rPPG methods often struggle in real-world conditions. This is largely due to a lack of diverse and ecologically valid benchmark datasets. Most current datasets are collected under bright lighting and at resting heart rates, failing to represent common scenarios like dim environments or situations where a person's heart rate is elevated due to physical activity, stress, or medical conditions. This underrepresentation leads to models that perform well on benchmarks but fail in practical applications. This paper aims to bridge this gap by systematically evaluating rPPG methods under these critical, yet underexplored, challenging conditions.

Methods

The researchers evaluated eight rPPG algorithms, comprising four signal processing-based methods (GREEN, CHROM, POS, ICA) and four deep learning-based methods (DeepPhys, TS-CAN, PhysNet, rPPGNet). They employed a two-pronged evaluation strategy:

  1. Within-dataset analysis: This involved training and testing models on the same dataset to assess their inherent robustness to challenging conditions.
  2. Cross-dataset generalization: Here, deep learning models were trained on existing public datasets and then tested on the newly introduced CHILL dataset to evaluate their ability to generalize to unseen challenging scenarios.

Performance was primarily measured using Mean Absolute Error (MAE) in beats per minute (bpm). Statistical significance was assessed using Wilcoxon signed-rank tests.

Datasets

  • PURE: A public rPPG dataset.
  • COHFACE: A public rPPG dataset.
  • CHILL (Challenging Heart Rate and Illumination): A newly introduced dataset collected by the authors. It comprises synchronized video and PPG signals from 45 participants under two lighting conditions (bright and dark) and two heart rate states (resting and elevated). This dataset is designed to specifically address the limitations of existing benchmarks by including more diverse participants and challenging recording conditions.

Key Insights and Ideas

  • Elevated heart rate is a major bottleneck: Unlike low illumination, which had a comparatively smaller impact, elevated heart rates consistently and significantly degraded the performance of most rPPG methods. This suggests that current algorithms are fundamentally less equipped to handle the physiological changes associated with higher heart rates.
  • Public benchmarks can be misleading: Deep learning models that perform exceptionally well on existing public datasets (PURE, COHFACE) often fail to generalize to the CHILL dataset, especially under challenging conditions. This highlights the risk of over-reliance on standard benchmarks and the need for more representative evaluation data.
  • Signal processing can still compete: Surprisingly, on the CHILL dataset, the signal processing-based method POS outperformed all deep learning methods, with CHROM also showing competitive performance. This suggests that for certain challenging scenarios, simpler, interpretable methods might be more robust than complex deep learning models.
  • Controlled vs. Uncontrolled Illumination Matters: While uncontrolled, fluctuating natural light poses a challenge (as seen with COHFACE), controlled bright and dark studio lighting (as in CHILL) showed that many methods could handle static low-light conditions reasonably well, suggesting the nature of illumination variation is critical.

Key Results

  • Within-dataset performance on CHILL: Five out of eight tested rPPG methods showed a statistically significant decline in performance at elevated heart rates. Low illumination had a less pronounced impact.
  • Generalization performance: Several deep learning models trained on public datasets failed to generalize effectively to the CHILL dataset. Even among those that did generalize, many still showed a significant performance drop under elevated heart rate conditions.
  • Specific method performance:
    • On CHILL, POS achieved the lowest MAE (1.1 ± 0.1 bpm), outperforming all deep learning methods.
  • Deep learning methods generally outperformed signal processing methods on PURE and COHFACE.
  • TS-CAN25 showed the most robust performance on CHILL when pre-trained on COHFACE, achieving the lowest MAEs across most conditions.
  • Heart rate impact: For models trained on PURE, DeepPhys, TS-CAN25, and PhysNet showed significant performance degradation at high heart rates. PhysNet exhibited a substantial median shift (-13.55 bpm), indicating high sensitivity.

Limitations and Future Work

  • Skin Tone Diversity: The CHILL dataset primarily includes participants with Fitzpatrick skin types I-III, limiting generalizability to individuals with darker skin tones. Future work should incorporate a broader range of skin tones.
  • Age Range: The dataset's age range (18-32 years) is limited. Future iterations should include older participants to account for age-related skin changes.
  • Artifacts: The current CHILL dataset does not capture motion artifacts or significant fluctuations in ambient lighting, which are common in real-world scenarios. Future work should aim to include these.
  • Video Length: The one-minute recording duration limits the analysis of heart rate variability and recovery dynamics. Longer recordings would enable more in-depth analysis.
  • Sensor Size: The study used a DSLR camera. Evaluating rPPG methods on smaller, ubiquitous sensors (smartphones, webcams) is crucial for practical deployment.
  • Statistical Power: The analysis of HR effects was limited to a subset of 21 participants, which could affect statistical power.
  • Evolving Methods: The field of rPPG is rapidly advancing, and new methods should be continuously evaluated against this dataset.

The authors emphasize the critical need for future research to prioritize validation across physiologically diverse scenarios, including elevated heart rates and other real-world challenges, to ensure reliable deployment of rPPG in digital health applications.

Summary updated

BibTeX
@article{acharya2025reliability,
  author = {Bhargav Acharya and William Saakyan and Barbara Hammer and Hanna Drimalla},
  title = {{The reliability of remote photoplethysmography under low illumination and elevated heart rates}},
  journal = {Npj Digit. Medicine},
  year = {2025},
  doi = {10.1038/s41746-025-02192-y},
}
Figure