Evaluation of video-based rPPG in challenging environments: Artifact mitigation and network resilience

Nhi Nguyen, Ngu Nguyen, Honghan Li, Miguel Bordallo López, Constantino Álvarez Casado

Abstract

Remote photoplethysmography (rPPG) enables non-contact heart rate monitoring by detecting subtle color variations in skin tissue via standard camera sensors. While effective in controlled laboratory settings, rPPG performance often degrades in real-world environments due to video compression, facial occlusions, and network instability. This paper systematically evaluates these challenges and proposes a suite of mitigation strategies, including GAN-based inpainting and signal reconstruction. Our findings demonstrate that while certain artifacts significantly impede accuracy, strategic signal processing and interpolation can maintain reliable monitoring even under severe network stress.

Introduction

Remote photoplethysmography (rPPG) represents a significant advancement in telemedicine and fitness tracking, offering a non-invasive method to measure cardiac cycles without physical sensors. By analyzing the light reflected from the skin, rPPG algorithms can extract the pulse signal. However, the transition from "goldilocks" laboratory conditions to real-world deployment introduces significant noise. Factors such as low-bandwidth video compression, facial masks, and packet loss during transmission can turn a clean physiological signal into unusable data. This study provides a comprehensive evaluation of these failures and explores the efficacy of modern computer vision techniques in rescuing the signal.

Methodology

The researchers developed a robust evaluation framework categorized into two primary modules: degradation and mitigation.

1. The Degradation Module

To simulate real-world conditions, clean video data was subjected to intentional degradation across four dimensions:

  • Spatial Factors: Reducing resolution and bit depth (down to 2-bit color).
  • Visual Artifacts: Applying Gaussian blur and additive noise.
  • Occlusions: Digitally simulating face masks and sunglasses using landmark-based warping to obscure signal-rich regions.
  • Temporal Variation: Simulating random frame loss (10%, 20%, and 50%) to mimic unstable network connections.

2. The Mitigation Module

Several strategies were tested to counteract these degradations:

  • Denoising: A comparison between traditional Non-Local Means (N-LM) filtering and a deep learning-based NAFNet model.
  • Inpainting: For occluded regions, the study compared a simple "Original Skin" (OS) method (masking the occlusion) against a Generative Adversarial Network (GAN) designed to reconstruct facial features.
  • Signal Reconstruction: Utilizing linear interpolation to bridge gaps caused by temporal frame loss.

These methods were evaluated across seven public datasets, including COHFACE, PURE, and UBFC-rPPG, covering a range of activities from rest to high-intensity exercise.

Key Insights and Discussion

The 6-Bit Threshold

A significant practical finding is that reducing color depth from 8-bit to 6-bit has a negligible impact on rPPG accuracy. This suggests that video codecs can employ aggressive compression without destroying the underlying physiological signal, which is critical for low-bandwidth telemedicine applications.

Impact of Occlusions

The study found a distinct difference between types of occlusions. Sunglasses have a minimal impact because the forehead and cheeks remain visible and provide sufficient signal. Conversely, face masks are highly disruptive, often doubling the error rate as they cover the most vascularized and signal-rich areas of the lower face.

The GAN Paradox

While GAN-based inpainting significantly improved visual metrics such as Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR), it did not consistently improve heart rate accuracy. This indicates that while GANs can "hallucinate" a realistic-looking face, they do not necessarily recover the authentic sub-pixel pulse signal. In many cases, the simpler "Original Skin" (OS) approach—focusing only on non-occluded areas—outperformed complex generative reconstruction.

Network Resilience

One of the most promising results involves temporal stability. By employing linear interpolation for missing frames, the researchers maintained reasonable accuracy even at 50% packet loss. This demonstrates that rPPG systems can be made resilient to the "jitter" common in standard video conferencing tools.

Conclusion

This evaluation serves as a vital stress test for rPPG technology. While deep learning models offer powerful tools for image enhancement, traditional signal processing methods (like POS) often show greater resilience to distribution shifts caused by environmental artifacts. For engineers building next-generation health-tech, the path forward involves developing rPPG-specific loss functions that prioritize physiological signal integrity over visual realism and utilizing robust interpolation to handle the inherent chaos of real-world networking.

Summary updated

Abstract

Highlights•Comprehensive evaluation of rPPG signals extracted using both learning-based and traditional methods.•Quantification of the impact of spatial, temporal, and visual occlusion artifacts on rPPG signal quality.•Proposal and evaluation of novel denoising and inpainting strategies to enhance rPPG accuracy in measuring heart rate under noisy and occluded conditions.•Comparison of various mitigation strategies to improve rPPG signals for computational and network issues.
BibTeX
@article{nguyen2024evaluation,
  author = {Nhi Nguyen and Ngu Nguyen and Honghan Li and Miguel Bordallo López and Constantino Álvarez Casado},
  title = {{Evaluation of video-based rPPG in challenging environments: Artifact mitigation and network resilience}},
  journal = {Computers in Biology and Medicine},
  year = {2024},
}
Figure