TL;DR
This paper introduces TURNIP, a novel modular pipeline for estimating heart rate and heart rate variability from facial videos. By combining a U-Net architecture with recurrent units and a sophisticated signal extraction process, TURNIP achieves state-of-the-art performance, even in the presence of significant motion and partial occlusions, making remote health monitoring more robust and reliable.
What Problem Does This Paper Address?
Remote health monitoring, particularly the estimation of vital signs like heart rate from video (imaging photoplethysmography or iPPG), holds immense promise for accessible and non-intrusive healthcare. However, existing methods often struggle with real-world challenges such as head motion, facial expressions, and partial occlusions, leading to inaccurate readings. Many deep learning approaches also act as "black boxes," making it difficult to understand why they succeed or fail. This paper aims to develop a robust, interpretable, and state-of-the-art iPPG system that can reliably estimate heart rate and its variability in the face of these common issues.
Methods
The proposed system, TURNIP, is a modular pipeline consisting of three key components:
- Face and Landmark Detection: This module uses a pre-trained detector to identify facial regions and key landmarks. Crucially, it incorporates a novel mechanism to detect and label "self-occluded" or out-of-frame landmarks. This information is used to alert downstream modules about potentially noisy or unreliable data points.
- Time-Series Extraction: This module defines 48 facial regions based on augmented landmarks (interpolated/extrapolated from the initial 68 to 145). For each region, it extracts the average pixel intensity. A key innovation here is the use of the Red-over-Green (R/G) color channel ratio for RGB videos, which empirically improves signal strength and reduces motion noise compared to using individual color channels. Regions associated with invisible landmarks are assigned a large out-of-range value to be ignored. The extracted signals are then filtered and normalized.
- TURNIP Pulse Signal Estimation: This is the core denoising and pulse signal reconstruction module. It's a U-Net-style architecture modified with Gated Recurrent Units (GRUs) in its skip connections. This combination allows the network to process spatio-temporal information (across pixels and time) while also incorporating temporal recurrence, effectively extending its receptive field to better capture the underlying pulse waveform even in noisy data. The network is trained using a loss function that maximizes the Pearson Correlation Coefficient between the predicted and ground-truth pulse signals. Data augmentation techniques like "SpeedUp" and "SlowDown" are also employed to improve robustness.
Datasets
The paper evaluates its method on three publicly available datasets:
- MMSE-HR [44]: Contains videos with subjects exhibiting various emotions, with ground-truth physiological data captured by a finger sensor. This dataset is noted for its significant head motion and occlusions.
- MR-NIRP Car [8]: Uses near-infrared (NIR) cameras and includes videos of subjects in a car, both driving and parked. This dataset presents challenges due to NIR frequency limitations and weaker signal-to-noise ratios.
- PURE [45]: Features subjects performing various head motion tasks, with synchronized video and pulse oximeter data.
The authors train and test on each dataset independently, and also conduct cross-dataset evaluations.
Key Insights and Ideas
- Modularity for Interpretability: Breaking down the iPPG pipeline into distinct, interpretable modules (detection, extraction, estimation) is a significant advantage over end-to-end black-box models. This allows for better understanding and debugging.
- Leveraging Landmark Visibility: Explicitly detecting and handling "self-occluded" landmarks is a novel and effective way to inform the system about unreliable data, acting as a form of implicit attention to improve robustness.
- R/G Ratio for RGB: Using the Red-over-Green intensity ratio for RGB videos is a simple yet powerful technique to enhance the pulsatile signal and mitigate motion artifacts, outperforming individual color channels.
- Recurrence in U-Nets for Temporal Robustness: Integrating GRUs into the U-Net's skip connections allows the model to better capture temporal dependencies, crucial for denoising noisy iPPG signals.
Key Results
TURNIP achieves state-of-the-art results across all tested datasets:
- MMSE-HR: Reduces Mean Absolute Error (MAE) from 1.84 bpm (previous state-of-the-art) to 1.17 bpm, and Root Mean Squared Error (RMSE) from 4.83 to 3.46 bpm. It also achieves a high PTE6 (Percent of Time that Error < 6 bpm) of 93.21%.
- PURE: Achieves an MAE of 0.36 bpm, RMSE of 0.67 bpm, and a perfect PTE6 of 100%, outperforming previous methods.
- MR-NIRP Car: Demonstrates significantly higher PTE6 than previous methods on both "Driving" and "Garage" subsets, indicating more accurate heart rate estimation within the 6 bpm tolerance.
- Pulse Rate Variability (PRV): The paper also shows promising results in estimating PRV metrics, which are important for assessing autonomic nervous system activity.
Limitations and Future Work
- Outliers in Bland-Altman Plots: While generally performing well, the Bland-Altman plots show a few outliers, particularly at higher heart rates or when training on PURE and testing on MMSE-HR. The authors suggest this might be due to insufficient training data at those rates or differences in dataset characteristics.
- Real-world Variations: The authors acknowledge that future work should continue to address real-world challenges like varying illumination and motion, aiming for even greater robustness.
- Reconstructing Waveform Characteristics: While heart rate estimation is strong, the authors suggest further research could focus on more effectively reconstructing the detailed characteristics of the pulse waveform itself, particularly for PRV analysis.