A phone camera can read your pulse. Each heartbeat changes facial skin color by about 1–2%, and a network that watches the video can recover that waveform, a technique called remote photoplethysmography (rPPG). Video codecs such as H.264 were designed to throw away exactly this kind of faint, fast flicker, because people cannot see it. The best existing fix chains a video-restoration network to a pulse extractor, both built from full 3D convolutions (filters that slide over space and time together). Its authors say it is too heavy to run in real time on a phone. We propose to search automatically for a single-stage pulse network. The search is trained on codec-degraded video across a sweep of bitrates, and it is penalized by latency and memory measured on real phones rather than by operation counts. The search space is organized around one decision: which kind of temporal operator each block uses, and how deep in the network the first one sits.
The scientific bet is that compression reverses what earlier searches found about where temporal operators belong. On clean video, architecture search for rPPG put temporal filters first and spatial filters later. On heavily quantized video, we expect temporal differencing of raw pixels to amplify the codec's blocky errors. The better design should first pool spatially across many coding blocks and apply temporal operators afterwards, which is closer to what action-recognition search finds. If this holds, it explains why the restoration stage was needed and shows how to remove it cheaply. If it fails, the experiments still give a measured map of which efficient video operators survive compression on real hardware.
Core Claim
Under lossy compression, the network layout that recovers pulse most accurately within a fixed on-device latency moves its first temporal operator deeper, behind learned spatial pooling. The mechanism is noise statistics. The pulse is spatially coherent across skin. Quantization error is largely confined to individual coding blocks, so it is only partly correlated from one block to the next. Averaging over $N$ blocks before looking at time should therefore raise pulse signal-to-noise ratio by up to roughly $\sqrt{N}$, the standard gain from averaging partly independent noise. Frame differencing on raw pixels instead doubles the quantization noise it sees and adds artificial edges at block boundaries. A search that sees compressed inputs and real latency should discover this layout on its own. The resulting network should match or beat the two-stage restoration pipeline at a fraction of its phone latency.
Where This Sits in Prior Work
Three papers motivate the gap. Remote Heart Rate Measurement From Highly Compressed Facial Videos pairs a video-enhancement network (STVEN) with rPPGNet to recover pulse from H.264/MPEG-4 video down to 250 kb/s. It is hand-designed and 3D-heavy, and the authors state that this blocks mobile deployment. We keep its compression protocol but drop the separate restoration stage. AutoHR runs gradient-based architecture search for rPPG and introduces Temporal Difference Convolution (TDC), which convolves frame-to-frame differences. It finds temporal operators early and spatial ones late. It never sees compressed video or hardware cost, and that combination is the one we add. Auto-X3D searches block-level 3D kernel choices for action recognition under a FLOP penalty. It finds 3D kernels most useful in middle and late layers. The disagreement between AutoHR and Auto-X3D on placement is the hypothesis we test. The claim is that compression is the variable that moves rPPG from one regime to the other.
Two lines supply the method. For hardware awareness we reuse per-operator latency lookup tables measured on the device and a one-shot weight-sharing supernet, as in MoGA. We also reuse the multi-constraint max-penalty loss of DUCCIO, which handles a latency limit and a memory limit without hand-tuning one weight per constraint. Neither method has been applied to spatiotemporal operators or to continuous waveform regression. For the operator menu we draw on the efficient-3D literature. Factorized spatio-temporal convolutions split a 3D kernel into a 2D spatial kernel followed by a 1D temporal one. Temporal shift moves a fraction of channels across time at zero FLOP cost, and that paper reports that shifting too many channels degrades spatial features. X3D shows that thin, high-resolution video networks beat wide ones.
Two sources constrain the design. HAT and the hardware-optimization survey document that FLOPs predict latency poorly, which is why we measure rather than count. Arbitrary-resolution rPPG shows that bilinear resizing itself corrupts pulse phase. We therefore fix the input crop geometry to align with codec block boundaries instead of resizing arbitrarily. The file also records that pre-averaging over fixed facial regions, as RhythmNet does, loses phase information under motion. Our pooling is learned and searched, not fixed. Whether it avoids that failure is tested directly on the motion splits.
Research Program
We build a supernet whose blocks each choose among six options. The first two are full 3D convolution and factorized (2+1)D convolution. The third is TDC. The fourth is a temporal shift at one of three ratios on a 2D convolution. The fifth is pure 2D convolution. The sixth is a learned spatial pooling operator with a pooling extent aligned to 8, 16 or 32 pixels. Widths and temporal kernel lengths are also searched. Temporal resolution is never subsampled, because heart-rate variability needs frame-level timing.
Training clips are re-encoded on the fly with H.264 and H.265 at bitrates sampled from near-lossless down to 250 kb/s and below. For each target device we measure every operator configuration's latency and peak activation memory. The devices are one mobile CPU and one mobile GPU, following MoGA's finding that the two platforms prefer different designs.
The search objective is
$$\min_{a}\; \mathbb{E}_{b \sim p(b)}\big[\mathcal{L}_{\text{pulse}}(a; b)\big] + \sum_{j} \lambda_j \max\big(0,\, R_j(a) - T_j\big).$$
Here $a$ is the architecture and $b$ is the codec bitrate drawn from the training distribution $p(b)$. $\mathcal{L}_{\text{pulse}}$ is negative Pearson correlation plus a power-spectrum term, as in PhysNet and AutoHR. $R_j(a)$ is the looked-up latency or memory of $a$, $T_j$ is its budget, and $\lambda_j$ is its penalty weight. Evolutionary search over the trained supernet then produces a latency–accuracy frontier per device and per bitrate regime.
Alongside the search we run a mechanism study. On OBF we measure the ground-truth-referenced pulse SNR of the codec-decoded skin signal as a function of pooling extent and bitrate. We compare the result to the $\sqrt{N}$ prediction and to raw frame differencing. This tells us, independently of any network, whether the predicted placement reversal has a physical basis.
Baselines
The naive reference is POS, run through rPPG-Toolbox on the same compressed clips. It costs almost nothing on device and, per MMPD, is surprisingly robust across lighting and skin tone. Any network that cannot clearly beat it at low bitrate has no deployment case.
The strongest accuracy competitor is STVEN + rPPGNet (Yu et al.). It is the only published method built for heavily compressed rPPG. It reports a cross-dataset RMSE of 5.93 bpm when trained on OBF and tested on MAHNOB-HCI. On OBF at 250 kb/s, the file records only that it beat CHROM and POS: no reported result found on Lacuna for its exact RMSE at that bitrate. Its phone latency is also unreported, so we will measure it ourselves. We re-implement it under the same bitrate sweep, subject split and training epochs. Beating it means equal or lower HR RMSE on OBF across the sweep and on the MAHNOB-HCI transfer, at no more than one fifth of its measured phone latency.
The strongest search competitor is AutoHR, which reports 8.68 bpm RMSE on VIPL-HR. We give it two treatments. The first searches as published on clean data. The second re-runs its search with our compression augmentation. Both are then retrained on compressed data with the same budget.
The most dangerous hand-built baseline is PhysNet converted to (2+1)D and shifted variants, width-scaled to our latency budget and trained with the identical codec augmentation. PhysNet reports 1.812 bpm RMSE on raw OBF (PhysNet). A FLOP-penalized version of our own search, in the style of Auto-X3D, isolates the value of measured latency.
The key ablation searches the identical space on clean video, then retrains the found architecture on compressed video. If it matches the compression-in-the-loop search, compression-aware search adds nothing. All searched and hand-built models get the same GPU-hour budget for search plus training, and all are timed on the same phones with the same runtime.
Why not just…?
Why not just train an efficient hand-built network with compression augmentation? This is the real threat. A factorized PhysNet sized to the latency budget and trained on re-encoded clips may capture most of the gain. We treat it as a primary baseline, not a strawman. If it comes within 0.5 bpm RMSE of the searched model at equal measured latency across bitrates, we will report that the search is unnecessary. The placement study would then stand as the contribution: we would transplant the searched operator ordering into the hand-built network and test whether ordering alone closes the gap.
Why not just use classical POS or CHROM on the phone? They are nearly free, but the file shows they break where it matters. Classical methods trail learned ones on VIPL-HR (16.9 vs 8.68 bpm RMSE for Haan2013 vs AutoHR). POS nearly doubles its MAE when subjects walk (9.70 to 17.05) (MMPD). STVEN + rPPGNet outperforms both at 250 kb/s. We still report POS on every split, because on stationary, well-lit, moderate-bitrate clips it may be good enough.
Why not just prepend a small denoiser to an existing pulse network? This keeps the two-stage structure that STVEN uses, only shrunk. Video-to-video restoration must output full-resolution frames, which costs memory at the resolution where pulse is weakest. Our mechanism study predicts that most of the useful restoration is spatial averaging the extractor can do internally. We include a latency-matched small restorer plus PhysNet as a baseline, and the comparison at equal phone latency settles it.
Why not just search with a FLOP penalty, as Auto-X3D does? Temporal shifts and factorized operators are cheap in FLOPs but costly in memory traffic on phones. That is the FLOPs–latency mismatch documented by HAT. The FLOP-penalized variant of our search tests this directly. If its frontier coincides with the latency-penalized one on both devices, measured tables are not needed for this operator family, and we will say so.
Proposed Experiments
The main benchmark is OBF re-encoded with H.264 and H.265 at five bitrates down to 250 kb/s, following the STVEN protocol. We report HR RMSE, MAE, Pearson correlation and HRV low- and high-frequency error. Cross-dataset transfer is tested on MAHNOB-HCI and on COHFACE, which is natively compressed and on which PFE-TFA reports 1.31 bpm MAE. Motion robustness is tested on VIPL-HR. Real phone capture comes from MMPD, broken down by skin type and activity, because the file reports that TS-CAN's MAE of 3.06 on skin type 3 rises to 15.43 on type 6. Every model is reported with measured median latency and peak memory on both devices.
Three checks decide the claim. First, the depth of the first temporal operator in the found architectures should increase monotonically as the training bitrate falls. Swapping the clean-searched and compressed-searched orderings, then retraining, should cost each model accuracy mainly in its own regime. Second, the network-free SNR measurement should show pooled-then-differenced signals beating differenced raw pixels at low bitrate, with gains near the $\sqrt{N}$ trend. A flat trend would falsify the mechanism even if the search results look good. Third, the ordering of candidates by measured phone latency should disagree with the ordering by FLOPs for the shift and factorized operators. If it does not, measured tables are not needed for this operator family.
References
- Remote Heart Rate Measurement From Highly Compressed Facial Videos: An End-to-End Deep Learning Solution With Video Enhancement
- AutoHR: A Strong End-to-End Baseline for Remote Heart Rate Measurement With Neural Searching
- Auto-X3D: Ultra-Efficient Video Understanding via Finer-Grained Neural Architecture Search
- MoGA: Searching Beyond Mobilenetv3
- Enhancing Neural Architecture Search With Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
- Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks
- Temporal shift reinforcement learning
- X3D: Expanding Architectures for Efficient Video Recognition
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
- Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
- Learning Motion-Robust Remote Photoplethysmography through Arbitrary Resolution Videos
- rPPG-Toolbox: Deep Remote PPG Toolbox
- MMPD: Multi-Domain Mobile Video Physiology Dataset
- Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks