TL;DR
This paper introduces ALPINE, a novel deep learning approach for remote heart rate (HR) estimation from facial videos. By leveraging contrastive learning and a hybrid loss function, ALPINE effectively mitigates the challenges posed by facial deformations and limited labeled data, achieving state-of-the-art performance.
What Problem Does This Paper Address?
Accurately measuring heart rate (HR) is crucial for monitoring cardiovascular health. Traditional methods like ECG and PPG require direct skin contact, which is problematic in scenarios like pandemics due to infection risks. Remote photoplethysmography (rPPG) offers a non-contact solution by analyzing subtle color or motion variations in facial videos. However, existing rPPG methods struggle with facial deformations (e.g., blinking, expressions, head movements) and often require vast amounts of labeled data for training, which is expensive and time-consuming to acquire. This paper aims to address these limitations by developing a robust and data-efficient rPPG method.
Methods
The core of ALPINE is a novel deep learning network called the Contrastive rPPG Network (CrPPG-Net), built upon the Temporal Convolutional Network (TCN) architecture. TCNs are chosen for their ability to effectively model long-term temporal dependencies in sequential data, which is crucial for capturing subtle pulse signals.
Key methodological contributions include:
- Contrastive Learning for Temporal Signals: ALPINE utilizes contrastive learning to learn similarities between temporal signals extracted from multiple facial regions (ROIs). This helps to suppress noise and facial deformation artifacts by focusing on consistent pulse information across different parts of the face.
- Hybrid Loss Function: A novel hybrid loss function is introduced, comprising three components:
- Contrastive Loss ($L_c$): Maximizes the similarity between denoised temporal signals from different facial ROIs, and also between augmented versions of these signals. This encourages robust feature learning.
- Signal-to-Noise Ratio (SNR) Loss ($L_{SNR}$): Aims to improve the quality of the denoised temporal signal by ensuring the HR frequency component is significantly higher than noise.
- Data Fidelity Loss ($L_{DF}$): Ensures that the predicted temporal signals closely match the ground truth pulse signals.
- ROI Extraction: Specific facial regions (left cheek, forehead, nose, and right cheek) are selected as ROIs. These regions are chosen because they are less susceptible to deformations compared to areas like the eyes or lips.
- Temporal Signal Extraction: Temporal signals are extracted from each ROI by averaging green channel intensity values across frames. These signals are then filtered to remove irrelevant frequency components.
- Pulse Signal and HR Estimation: During inference, denoised temporal signals from all ROIs are consolidated using the Multi-Kurtosis Optimization method to extract the pulse signal, from which the HR is computed via FFT.
Datasets
The proposed method was evaluated on two publicly available datasets:
- UBFC-rPPG: Contains 42 face videos with corresponding ground truth physiological signals.
- COHFACE: Consists of 160 face videos with ground truth physiological signals.
For both datasets, a standard train-test split (67%/33% for UBFC-rPPG and 60%/40% for COHFACE) was used, consistent with prior work.
Key Insights and Ideas
- Leveraging Multiple ROIs is Crucial: By extracting temporal signals from multiple facial regions, ALPINE can effectively mitigate the impact of local noise and deformations that might affect a single ROI. This distributed approach enhances robustness.
- Contrastive Learning for Temporal Data is Powerful: Applying contrastive learning not just to spatial features but to temporal signals from different facial regions is a key innovation. It allows the model to learn what constitutes "pulse signal" versus "noise" without explicit labels.
- Hybrid Loss for Comprehensive Learning: The combination of contrastive, SNR, and data fidelity losses provides a well-rounded training objective. Contrastive loss handles robustness to deformations, SNR loss improves signal quality, and data fidelity loss ensures accuracy against ground truth.
- TCNs Excel at Temporal Dependencies: The choice of TCNs over recurrent networks like LSTMs is justified by their superior ability to model long-range temporal dependencies, which is essential for accurate pulse extraction from subtle video signals.
Key Results
ALPHINE demonstrated state-of-the-art performance on both UBFC-rPPG and COHFACE datasets, outperforming existing methods across various metrics including Mean Absolute Error (MAE), Standard Deviation (SD), Pearson's correlation coefficient (r), and Root Mean Squared Error (RMSE). For instance, on the UBFC-rPPG dataset, ALPINE achieved an MAE of 2.58 BPM and an r value of 0.96, and on COHFACE, an MAE of 3.65 BPM and an r value of 0.82. Ablation studies confirmed the importance of multiple ROIs, the contrastive loss, and the green color channel for optimal performance.
Limitations and Future Work
- Limitations: The paper acknowledges that ALPINE can still produce incorrect HR estimates when input clips contain significant deformations across all ROIs simultaneously, as the model might learn noise along with pulse signals in such extreme cases.
- Future Work: The authors suggest exploring transformer-based networks in unsupervised settings for future rPPG computation. This could potentially further enhance the model's ability to capture long-range dependencies and generalize to diverse scenarios.
Summary updated