PhysBench: A Benchmark Framework for Remote Physiological Sensing with New Dataset and Baseline

Kegang Wang, Yantao Wei, Mingwen Tong, Jie Gao, Yi Tian, YuJian Ma, ZhongJin ZhaoarXiv 2023 · DOI

TL;DR

This paper introduces the RLAP dataset, a highly synchronized collection of remote photoplethysmography (rPPG) data specifically designed for remote learning environments. They also propose Seq-rPPG, a novel 1D CNN-based model that achieves state-of-the-art performance in Heart Rate Variability (HRV) tasks with significantly reduced computational overhead, making it suitable for mobile deployment.

What Problem Does This Paper Address?

Remote photoplethysmography (rPPG) aims to extract physiological signals, like heart rate (HR) and heart rate variability (HRV), from facial videos. While rPPG has advanced significantly, a major bottleneck for accurate HRV analysis has been the lack of precisely synchronized ground truth physiological signals in existing datasets. Many datasets rely on separate devices for video and physiological data, leading to synchronization errors that can range from 100 to 200 milliseconds. These errors are critical because HRV analysis operates on a timescale of tens of milliseconds, making existing datasets unsuitable for training robust HRV models. Furthermore, compressed video formats can degrade the quality of the BVP signals, necessitating lossless data collection. This paper addresses these limitations by creating a new, highly synchronized dataset and a model optimized for HRV analysis.

Methods

The core of the paper's methodological contribution lies in two areas: the dataset collection and the proposed model.

RLAP Dataset Collection: The authors meticulously collected the Remote Learning Affect and Physiology (RLAP) dataset. Key aspects include:

  • High Synchronization: They developed a custom tool, PhysRecorder, to simultaneously capture video and pulse oximeter data, ensuring precise alignment of BVP signals with video frames.
  • Lossless Video: They prioritized lossless video formats (MJPG and YUY2) to preserve signal quality.
  • Diverse Scenarios: The dataset captures subjects in various remote learning-related scenarios, including general relaxation, emotional induction tasks (watching videos), and learning engagement tasks.
  • Large Scale: It comprises over 32 hours of synchronized data from 58 subjects.

Seq-rPPG Model: The proposed Seq-rPPG model is a 1D Convolutional Neural Network (CNN) designed for temporal feature extraction.

  • Encoder-Decoder Architecture: It follows an encoder-decoder structure, where the encoder processes RGB channels and frame numbers into a coarse signal.
  • Spectral Transformation Module: A key innovation is the integration of a spectral transformation module, inspired by Fast Fourier Convolution (FFC), which enhances the model's ability to process periodic signals like BVP. This module alternates with temporal domain CNN layers.
  • Focus on Temporal Features: Unlike many spatio-temporal models, Seq-rPPG prioritizes temporal features, allowing for longer receptive fields and better capture of periodic HRV characteristics.
  • Computational Efficiency: The 1D CNN structure and efficient encoding contribute to significantly lower computational overhead compared to existing methods.

Datasets

  • RLAP Dataset (New): Introduced in this paper, it's designed for remote learning and affective computing, featuring highly synchronized video and BVP labels from 58 subjects over 32.7 hours.
  • UBFC-rPPG [1]: Used for cross-dataset testing, it contains 42 subjects with 1-minute video clips.
  • PURE [28]: Also used for cross-dataset testing, it includes 59 videos from 10 subjects performing various head movements.

Key Insights and Ideas

  • Synchronization is Paramount for HRV: The paper strongly reinforces that precise synchronization between video and ground truth physiological signals is not just beneficial but essential for accurate HRV analysis. Even small offsets can significantly degrade performance.
  • RLAP Dataset's Value: The RLAP dataset's diverse scenarios, high synchronization, and lossless video format make it a superior training resource for HRV tasks compared to existing datasets like UBFC-rPPG.
  • 1D CNNs Excel at Temporal Tasks: The success of Seq-rPPG highlights that for HRV analysis, focusing on temporal feature extraction with 1D CNNs can be more effective and computationally efficient than complex spatio-temporal models.
  • Longer Time Windows are Crucial for HRV: The study demonstrates that a larger temporal receptive field (up to 450 frames or 15 seconds) significantly improves HRV estimation, particularly in handling noise and distortions from head movements.
  • Mobile-Friendly rPPG is Achievable: Seq-rPPG's low computational overhead (approximately 1/30th of comparable models) makes it a strong candidate for real-time deployment on mobile devices, democratizing access to physiological monitoring.

Key Results

  • Intra-dataset Performance: Seq-rPPG achieved state-of-the-art performance on the RLAP dataset for HRV tasks (SDNN, pNN50, RMSSD), outperforming all baselines. While HR MAE was competitive, RMSE was slightly behind TS-CAN and PhysNet.
  • Cross-dataset Performance: Seq-rPPG consistently led in all tasks (HR and HRV) across UBFC-rPPG and PURE datasets when trained on RLAP, demonstrating superior generalization and transferability.
  • Computational Efficiency: Seq-rPPG exhibited significantly lower computational overhead, with an average frame time of 0.36 ms on a Raspberry Pi 4B, compared to models like PhysNet (9.69 ms) and TS-CAN (10.72 ms).
  • Time Window Impact: Increasing the time window for Seq-rPPG from 100 to 450 frames led to a significant reduction in SDNN error, with diminishing returns beyond 400 frames.

Limitations and Future Work

  • Limitations:
    • The paper notes that while Seq-rPPG's HR RMSE was slightly behind TS-CAN and PhysNet in intra-dataset testing, its overall advantage in HRV and computational efficiency is more significant.
  • The study primarily focuses on Chinese student subjects, and further validation on more diverse populations might be beneficial.
  • Future Work:
    • The authors suggest that the RLAP dataset's rich emotional and engagement data could be leveraged for more advanced affective computing applications.
  • The lightweight nature of Seq-rPPG opens avenues for real-time, on-device physiological monitoring in various applications, including remote learning and healthcare.
  • Further exploration of the spectral transformation module and its integration with different network architectures could yield further improvements.

Summary updated

BibTeX
@misc{wang2023physbench,
  author = {Kegang Wang and Yantao Wei and Mingwen Tong and Jie Gao and Yi Tian and YuJian Ma and ZhongJin Zhao},
  title = {{PhysBench: A Benchmark Framework for Remote Physiological Sensing with New Dataset and Baseline}},
  year = {2023},
  eprint = {2305.04161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.04161},
}
Figure