When your heart beats, it pumps blood throughout your body, causing rhythmic volume changes in the blood vessels of your face. Although invisible to the naked eye, these changes alter the amount of light reflected by your skin. Remote photoplethysmography (rPPG) is the science of using standard webcams or smartphone cameras to detect these minute color fluctuations, allowing for non-contact monitoring of heart rate and heart rate variability (HRV).
However, deploying rPPG in the real world reveals a fundamental engineering conflict: the very algorithms we use to transmit and store video across the internet are mathematically designed to destroy the exact signals rPPG relies on.
The Perceptual Coding Conflict
To understand why video compression breaks rPPG, we have to look at how modern video codecs work. Codecs like H.264 (x.264) and H.265 (x.265) are built around perceptual coding. Their primary goal is to minimize file size while keeping the video visually indistinguishable to a human observer.
As Christian S. Pilz et al. point out in Local Group Invariance for Heart Rate Estimation From Face Videos in the Wild, human vision is highly insensitive to high-frequency, low-amplitude spatial and temporal color variations. Consequently, compression algorithms intentionally discard this "redundant" data.
Unfortunately, as Jeremy Speth et al. note in Non-Contrastive Unsupervised Learning of Physiological Signals from Video, the blood volume pulse manifests as a tiny 1–2% variation in pixel intensity. Because this physiological signal lives entirely within the low-amplitude, high-frequency spectrum that codecs discard, standard compression treats the human heartbeat as visual noise and systematically erases it.
[Raw Video] ---> [Lossy Codec (Quantization & Smoothing)] ---> [Compressed Video]
Contains Discards low-amplitude, high-frequency Heartbeat signal
1-2% pulse color variations to save bandwidth is flattened or
fluctuations heavily distorted
How Compression Destroys the Pulse
Compression degrades the physiological signal through two primary mechanisms: spatial-temporal smoothing and quantization.
Spatial-Temporal Smoothing
Codecs group similar pixels into blocks and average them over time to save bandwidth. Radim Spetlík et al. demonstrate in Visual Heart Rate Estimation with Convolutional Neural Network that this high-level smoothing acts as a severe bottleneck for rPPG. By blending minute pixel-level variations across space and time, the codec flattens the subtle temporal waveforms required for pulse detection. Amogh Gudi et al. echo this in Efficient Real-Time Camera Based Estimation of Heart Rate and Its Variability, showing that this smoothing makes it incredibly difficult for traditional signal processing methods to extract accurate HRV metrics, which require precise timing between beats.
Quantization Artifacts
Quantization maps a continuous range of color values into a smaller set of discrete steps. When a video is heavily compressed, the step size becomes larger than the 1–2% pulse signal itself. As a result, the subtle gradient of a heartbeat is rounded off into static blocks. John Gideon et al. explain in The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video that these quantization artifacts introduce artificial edges and distortions that completely obscure the underlying blood volume pulse.
The mathematical reality of this degradation is stark. In iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox, Ethan B. Blackford demonstrates that standard video codecs cause a linear degradation in the signal-to-noise ratio (SNR) of recovered blood volume pulse signals. As bitrate drops, the physiological signal vanishes in a predictable, destructive trajectory.
The Hidden Culprit: Spatial Interpolation
The loss of signal is not exclusively driven by video codecs; standard preprocessing pipelines also introduce corruption. To feed facial videos into deep neural networks, researchers typically standardize input dimensions by resizing frames.
However, in Learning Motion-Robust Remote Photoplethysmography through Arbitrary Resolution Videos, Jianwei Li et al. reveal that simple spatial interpolation algorithms (like bilinear or bicubic resizing) inadvertently corrupt high-frequency physiological cues embedded in pixel data. When pixels are mathematically blended to fit a new resolution, the delicate, localized phase information of the blood flow is blurred, compounding the damage already inflicted by lossy compression.
Downstream Bottlenecks for Machine Learning
This signal degradation creates severe roadblocks for modern deep learning approaches to rPPG, particularly in three areas:
- Unsupervised Learning: Large-scale, self-supervised models rely on massive, uncurated datasets (like web videos) to learn representations without medical ground truth. However, Jeremy Speth et al. note that because web content is heavily compressed, the physiological signal is often entirely missing from the source data, preventing unsupervised models from learning meaningful features. Similarly, H. Wang et al. show in Self-Supervised Representation Learning Framework for Remote Physiological Measurement Using Spatiotemporal Augmentation Loss that high compression combined with small dataset sizes masks the high-fidelity temporal information needed for robust signal reconstruction.
- Deepfake Detection: Security frameworks often use rPPG to detect face forgeries, operating on the premise that AI-generated faces lack a real biological pulse. However, Jiahao Liang et al. demonstrate in Identifying Rhythmic Patterns for Face Forgery Detection and Categorization that the heavy compression applied by social media platforms destroys these rhythmic patterns. This makes it incredibly difficult to distinguish a compressed real video from a compressed fake video based on physiology alone.
- The Real-World Performance Gap: Models trained on pristine, uncompressed laboratory data experience a sharp decline in accuracy when deployed in real-world settings. As Puneet Gupta et al. observe in MOMBAT: Heart Rate Monitoring from Face Video using Pulse Modeling and Bayesian Tracking, this creates a massive performance gap between high-quality reference streams and the lower-quality, compressed video streams typical of telehealth or mobile applications.
Engineering Around the Compression Barrier
To bridge this gap, recent research focuses on building architectures that are explicitly aware of compression artifacts.
One prominent strategy is neural video enhancement. In Remote Heart Rate Measurement From Highly Compressed Facial Videos: An End-to-End Deep Learning Solution With Video Enhancement, Wei Peng et al. introduce a Spatio-Temporal Video Enhancement Network (STVEN) paired with a specialized recovery network (rPPGNet). Instead of trying to extract a pulse from a degraded video directly, the framework uses a learned upsampling and de-blocking step to mathematically "reconstruct" the discarded low-amplitude color variations before performing rPPG extraction.
[Compressed Video] ---> [STVEN (Enhancement)] ---> [Restored Cues] ---> [rPPGNet] ---> [Accurate HR]
Alternatively, researchers are advocating for a shift in how data is captured. In introducing the MMPD: Multi-Domain Mobile Video Physiology Dataset, Jiankai Tang et al. emphasize that standard mobile video compression remains a primary barrier to non-contact health monitoring. They suggest that future breakthroughs will rely on developing specialized, physiology-preserving codecs or utilizing raw, uncompressed video formats directly at the point of capture.
Key Foundational Work
Visual Heart Rate Estimation with Convolutional Neural Network (2018) — contributed 3 concept elements to this direction.
iPhys: An Open Non-Contact Imaging-Based Physiological Measurement Toolbox (2019) — contributed 3 concept elements to this direction.
Local Group Invariance for Heart Rate Estimation From Face Videos in the Wild (2018) — contributed 2 concept elements to this direction.