WPPG Net: A Non-contact Video Based Heart Rate Extraction Network Framework with Compatible Training Capability

Weiyu Sun, Xinyu Zhang, Ying Chen, Yun Ge, Chunyu Ji, Xiaolin HuangarXiv 2022 · DOI · arXiv

Introduction

Remote photoplethysmography (rPPG) is a non-contact method for measuring heart rate by detecting subtle skin color changes in facial videos. Despite significant advancements driven by deep learning, the field faces a persistent "ground truth" crisis. This crisis stems from the inherent temporal misalignments between video data and traditional medical sensors, which complicates the training of end-to-end networks.

The Ground Truth Crisis

Standard rPPG datasets typically utilize Electrocardiogram (ECG) or finger-clip Blood Volume Pulse (BVP) sensors as labels. However, an uncertain temporal delay exists between the heart's electrical activity and the blood flow changes visible on the face. Furthermore, the "envelope shape" of these label waves varies significantly between sensors and individuals.

Temporal delay between physiological signals (ECG/BVP) and video input.

Traditionally, researchers have relied on manual alignment—a process that is time-consuming, error-prone, and often results in discarding a large portion of available data that cannot be perfectly synchronized. This bottleneck limits the utility of existing datasets and the scalability of rPPG models.

The BYHE Framework: Rhythm over Alignment

The proposed WPPG Net, utilizing the BYHE (Boost Your Heartbeat Estimation) framework, addresses this bottleneck by shifting the focus from absolute signal alignment to intrinsic rhythm self-similarity. Instead of attempting to predict a perfectly aligned 1D wave, the network focuses on the periodicity of the signal.

Methodology

The BYHE framework introduces a novel approach to label representation and loss calculation:

  1. Phase Map Representation: The ground truth is preprocessed using a Butterworth filter and Continuous Wavelet Transform (CWT). A Hilbert transform extracts the instantaneous phase to generate a Target Label Matrix ($R$). Each element $R_{i,j}$ represents the cosine similarity of the phase between two time points, making the representation invariant to constant temporal shifts.
  2. Network Architecture: The framework is designed as a "graft" that can be added to standard backbone networks (such as the Contrastive Attention Network). It adds an adjustment layer that uses sliding windows to extract feature vectors and calculates their cosine similarity to produce a predicted matrix ($\hat{R}$) that mirrors the structure of the label matrix.
The WPPG Net (BYHE) framework architecture, highlighting the self-similarity matrix and loss functions.

Loss Functions

Training is guided by a composite loss function designed to ensure rhythmic stability:

  • $Loss_{MSE}$: Ensures the predicted matrix matches the structure of the label matrix.
  • $Loss_{Pearson}$: Maintains rhythm consistency across rows of the matrix.
  • $Reg$ (Regularization): A novel term that encourages "diagonal uniformity," ensuring the detected rhythm remains stable over time and preventing the network from collapsing into non-rhythmic predictions.

Experimental Results

The BYHE framework was evaluated on two gold-standard datasets: VIPL-HR and MAHNOB-HCI. By ignoring the alignment struggle, the method utilizes nearly 100% of the dataset, compared to the ~50% often used by traditional methods that require clean, synchronized slices.

  • VIPL-HR: Achieved a Mean Absolute Error (MAE) of 4.85 bpm, outperforming specialized models like PhysFormer (4.97 bpm) and Dual-GAN (4.93 bpm).
  • MAHNOB-HCI: Achieved an MAE of 3.54 bpm, demonstrating competitive performance against state-of-the-art meta-learning and transformer-based approaches.

Ablation studies confirmed that the regularization term ($reg$) is vital; without it, the network fails to maintain a stable rhythmic signal.

Limitations and Future Work

While BYHE effectively solves the alignment problem, it remains sensitive to extreme noise. In scenarios with fierce motion or lighting changes that completely mask the rPPG signal, the self-similarity matrix becomes unreliable. Future research will focus on hybrid architectures and more precise facial skin masking to further isolate the rPPG signal from environmental noise.

Conclusion

The WPPG Net (BYHE) demonstrates that for heart rate estimation, the interval between beats is more critical than the exact timestamp. By utilizing self-similarity matrices, the network learns the "tempo" of the heart without the confusion caused by sensor delays. This approach significantly improves data efficiency and allows classical architectures to achieve state-of-the-art performance in remote physiological monitoring.

Summary updated

Abstract

Heart rate measuring based on remote photoplethysmography (rPPG) plays an important role in health caring, which estimates heart rate from facial video in a non-contact, less-constrained way. End-to-end neural network is a main branch of rPPG-based heart rate estimation methods, whose trait is recovering rPPG signal containing sufficient heart rate message from original facial video directly. However, there exists some easily neglected problems on relevant datasets which thwarting the efficient training of end-to-end methods, such as uncertain temporal delay and indefinite envelope shape of label waves. Although many novel and powerful networks are proposed, hitherto there are no systematic research digging into these problems. In this paper, from perspective of common intrinsic rhythm periodical self-similarity results from cardiac activities, we propose a comprehensive methodology, Boost Your Heartbeat Estimation (BYHE), including new label representations, corresponding network adjustments and loss functions. BYHE can be easily grafted on current end-to-end network and boost its training efficiency. By applying our methodology, we can save tremendous time without conducting laborious handworks, such as label wave alignment which is necessary for previous end-to-end methods, and meanwhile enhance the utilization on datasets. According to our experiments, BYHE can leverage classical end-to-end network to reach competitive performance against those state-of-the-art methods on mostly used datasets. Such improvement indicates selecting perspicuous and efficient label representation is also a promising direction towards better remote physiological signal measurement.
BibTeX
@misc{sun2022wppg,
  author = {Weiyu Sun and Xinyu Zhang and Ying Chen and Yun Ge and Chunyu Ji and Xiaolin Huang},
  title = {{WPPG Net: A Non-contact Video Based Heart Rate Extraction Network Framework with Compatible Training Capability}},
  year = {2022},
  eprint = {2207.01697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.01697},
}
Figure