Abstract
Remote physiological measurement (RPM) through facial video analysis typically relies on large-scale datasets paired with gold-standard medical labels, such as ECG. However, acquiring such labels is resource-intensive and requires specialized equipment. This paper introduces SLF-RPM, a self-supervised learning (SSL) framework designed to estimate physiological signals like heart rate from unlabeled facial videos. By employing domain-specific spatiotemporal augmentations that respect the physics of skin reflection and signal sampling, SLF-RPM extracts meaningful heart rate features without human annotation. Experimental results demonstrate that SLF-RPM achieves accuracy competitive with fully supervised state-of-the-art methods across multiple public datasets.
Introduction
Remote photoplethysmography (rPPG) detects subtle, periodic color changes in the skin caused by blood flow using standard cameras. While deep learning has advanced this field, most models are "data-hungry," requiring expensive medical equipment to generate ground-truth labels.
Standard Self-Supervised Learning (SSL) techniques, such as random cropping or color jittering, are often destructive in the context of rPPG. These augmentations can eliminate the delicate physiological signals necessary for heart rate estimation. SLF-RPM addresses this by introducing "physiologically-aware" augmentations that preserve and highlight these signals, bridging the gap between unsupervised and supervised learning in remote physiological monitoring.
Methods: The SLF-RPM Framework
The core innovation of SLF-RPM lies in its two-pronged augmentation strategy, which is integrated into a contrastive learning pipeline using a 3D ResNet video encoder.
1. Landmark-Based Spatial Augmentation
Instead of arbitrary cropping, the framework utilizes facial landmarks to isolate seven specific Regions of Interest (ROIs), such as the forehead and cheeks. This approach is grounded in the Shafer dichromatic reflection model, which distinguishes between specular reflection (glare) and diffuse reflection (the light carrying the pulse signal). By focusing on these ROIs, the model learns to prioritize skin areas that provide the strongest physiological signals while ignoring background noise.
2. Sparsity-Based Temporal Augmentation
Based on the Nyquist–Shannon sampling theorem, the authors vary the frame rate (stride) of video clips. Since a heart rate signal can be reconstructed if the sampling rate is at least twice the signal's frequency, using different strides creates valid but distinct "views" of the same physiological event. This encourages the model to be invariant to frame rate variations common in real-world camera hardware.
3. Spatiotemporal Loss and Pseudo-labels
The system doesn't just use contrastive loss; it also generates "pseudo-labels" based on the specific ROI and temporal stride used. A secondary task of predicting these labels helps the model better characterize different facial regions and handle complex noise.
Results and Evaluation
The framework was evaluated on three benchmark datasets: MAHNOB-HCI, VIPL-HR-V2, and UBFC-rPPG. SLF-RPM consistently outperformed existing self-supervised methods and showed remarkable competitiveness with supervised models.
Key findings include:
- Superior SSL Performance: On the MAHNOB-HCI dataset, SLF-RPM achieved a Mean Absolute Error (MAE) of 3.60 bpm, significantly better than the 7.03 bpm achieved by the SeCo baseline.
- Closing the Supervision Gap: On the UBFC-rPPG dataset, SLF-RPM reached an MAE of 8.39 bpm, nearly matching the supervised baseline of 8.08 bpm.
- Enhanced Transferability: Pre-training on the large VIPL-HR-V2 dataset using SLF-RPM resulted in better transfer performance to smaller datasets compared to pre-training on general action recognition datasets like Kinetics-700.
Discussion and Conclusion
The success of SLF-RPM highlights the importance of grounding SSL augmentations in the physics of the target domain. By replacing random augmentations with physiologically-aware strategies, the model learns robust features that are invariant to hardware differences and environmental noise. While performance on the challenging VIPL-HR-V2 dataset suggests that further work is needed to handle extreme motion and lighting, SLF-RPM represents a significant step toward label-efficient remote physiological monitoring.
Summary updated