RADIANT: Better rPPG Estimation Using Signal Embeddings and Transformer

Anup Kumar Gupta, Rupesh Kumar, Lokendra Birla, Puneet GuptaWACV 2023

TL;DR

This paper introduces RADIANT, a novel non-contact heart rate estimation method that leverages signal embeddings and a Transformer architecture to overcome limitations of existing rPPG techniques. By effectively capturing long-term temporal dependencies and suppressing noise through attention mechanisms and specialized embeddings, RADIANT achieves state-of-the-art performance on public datasets.

What Problem Does This Paper Address?

Estimating heart rate (HR) non-invasively from facial videos, known as remote photoplethysmography (rPPG), holds significant promise for health monitoring and various applications. However, existing deep learning-based rPPG methods struggle with several challenges:

  • Noise Sensitivity: Subtle skin color variations indicative of pulse are easily corrupted by noise from facial movements, illumination changes, and other artifacts.
  • Local Feature Limitations: Convolutional Neural Networks (CNNs), commonly used in rPPG, tend to focus on local features, making them susceptible to noise in specific facial regions.
  • Long-Term Dependencies: Standard CNNs and even some sequential architectures fail to effectively model the long temporal dependencies crucial for accurate pulse signal extraction.
  • Data Scarcity: Training robust deep learning models often requires large datasets, which are not always readily available for rPPG, leading to underfitting.

This paper aims to address these issues by proposing a more robust and accurate rPPG estimation method.

Methods

RADIANT introduces a multi-stage approach:

  1. Temporal Signal Acquisition:
    • ROI Extraction: Facial landmarks are used to define specific Regions of Interest (ROIs) on the face, focusing on areas less prone to noise from expressions (e.g., avoiding eyes and forehead). These ROIs are further refined by selecting only skin pixels within these regions.
  2. Temporal Signal Extraction: RGB signals are extracted from the selected ROIs by averaging pixel values. These signals are then filtered using a bandpass filter to isolate frequencies within the normal HR range and projected into chrominance signals to reduce noise and specular reflections. A detrending filter is applied to mitigate illumination variations.
  3. Signal Embeddings:
    • Temporal Signal Selection: To handle varying numbers of ROIs across different video clips, a fixed number of temporal signals are selected. The method prioritizes signals with lower standard deviation, assuming they are less affected by facial movements and noise.
  4. MLP Projection: A Multilayer Perceptron (MLP) is used to project the selected temporal signals into higher-dimensional "signal embeddings." This step is crucial for creating a representation subspace that facilitates learning relevant rPPG features and aids in denoising. A learnable "classification token" embedding is prepended to these embeddings.
  5. Transformer Architecture:
    • The core of RADIANT is a Transformer architecture, known for its ability to model long-range dependencies through self-attention mechanisms.
  6. Multi-Head Self-Attention (MSA): This mechanism allows the model to weigh the importance of different signal embeddings when consolidating feature information, effectively learning contextual dependencies.
  7. Feature Consolidation and Denoising: The Transformer layers process these embeddings to consolidate rPPG information and denoise the signal. A two-layer MLP with GELU activation is used for feature transformation, and residual connections are employed.
  8. Pulse Signal Estimation: The output from the Transformer is fed into an MLP head that transforms it into the final pulse signal, matching the dimensions of the ground truth for training.
  9. Pre-training and Data Augmentation:
    • Pre-training: To combat the Transformer's lack of inductive bias and improve generalization, the model is pre-trained in two stages: first on the ImageNet dataset, and then on synthetic temporal signals that mimic periodic cardiovascular pulses, breathing rhythms, and noise.
  10. Data Augmentation: To address the uneven distribution of HR ranges in existing datasets, temporal signal interpolation is used to generate samples with higher and lower HR values, expanding the training data's coverage.

Datasets

The paper evaluates RADIANT on two publicly available datasets:

  • UBFC-rPPG: Contains 2-minute facial videos from 42 subjects, recorded at 640x480 resolution and 30 fps. The dataset is split into training (28 subjects) and testing (14 subjects) sets.
  • COHFACE: Includes 1-minute face videos from 40 subjects with physiological information, recorded at 20 fps.

The authors also generated synthetic temporal signals for pre-training, which are described as being generated using sine waves with frequencies between 0.7 Hz and 4.2 Hz, superimposed with breathing rhythms, random step signals, and Gaussian noise.

Key Insights and Ideas

  • Signal Embeddings for Noise Resilience: The proposed signal embeddings, generated by projecting temporal signals through an MLP, are key to isolating rPPG information. The selection of signals with lower standard deviation and the embedding process itself help to suppress noise and prevent corrupted signals from unduly influencing the final estimation.
  • Transformer's Strength in Temporal Modeling: The paper effectively leverages the Transformer's self-attention mechanism to capture long-term temporal dependencies, a critical weakness in many prior CNN-based rPPG methods. This allows for a more holistic understanding of the pulse signal over time.
  • Pre-training and Augmentation for Generalization: The two-stage pre-training strategy and the temporal signal augmentation techniques are shown to be vital for improving the model's generalization capabilities, especially in addressing the common issue of underfitting with limited or imbalanced datasets.
  • Quality Score for Signal Selection: Using the standard deviation of temporal signals as a quality score to select the most reliable signals is an intuitive and effective way to filter out noisy inputs before they reach the more complex processing stages.

Key Results

  • RADIANT achieves state-of-the-art performance on both the UBFC-rPPG and COHFACE datasets, outperforming several previous well-known rPPG methods.
  • Quantitative results (reported in Tables 1 and 2) show significantly lower Mean Absolute Error (MAE), standard deviation ($\sigma$), and Root Mean Squared Error (RMSE) compared to baseline methods. For instance, on UBFC-rPPG, RADIANT achieves an MAE of 03.45 and RMSE of 04.52, which is considerably better than many cited methods.
  • An ablation study (Table 2) indicates that using 12 ROIs provides optimal performance, with efficacy decreasing when more ROIs are included, suggesting a trade-off between capturing more information and introducing more noise.
  • The paper demonstrates that pre-training with synthetic data significantly improves performance, highlighting the effectiveness of this strategy for domain adaptation.

Limitations and Future Work

  • Limitations: The paper acknowledges that while their method is robust, slight noise due to facial movements can still cause deviations in estimated pulse signals for certain frames. The performance is also sensitive to the quality of the initial facial landmark detection.
  • Future Work: The authors plan to work on developing an end-to-end HR estimation network by automating the temporal signal extraction process. This suggests a move towards a more streamlined and fully integrated system.

Summary updated

BibTeX
@inproceedings{gupta2023radiant,
  author = {Anup Kumar Gupta and Rupesh Kumar and Lokendra Birla and Puneet Gupta},
  title = {{RADIANT: Better rPPG Estimation Using Signal Embeddings and Transformer}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure