Teaching rPPG to Walk: Pulse-Preserving Translation of Still Faces into Walking Video

Cameras can read a person's heart rate from video. Each heartbeat changes the skin's color very slightly, and remote photoplethysmography (rPPG) recovers that color signal. The method works well when the subject sits still and fails when they walk. On the MMPD smartphone benchmark, the training-free POS method degrades from 9.70 to 17.05 bpm mean absolute error (MAE) when subjects walk, and the TS-CAN network exceeds 25 bpm (MMPD). The obvious fix is to train on labelled walking video. That data is scarce, because the contact sensors that provide ground truth themselves slip during motion, and walking cadence falls in the same 1–3 Hz band as the heart rate.

We propose to manufacture that data. Labelled stationary videos are plentiful. We would translate them into realistic walking videos while keeping the original, known pulse intact, then train ordinary supervised estimators on the result. The translator is learned without paired examples, using unlabelled real walking footage only as a style target. The technical crux is temporal consistency. A standard anti-flicker loss penalizes frame-to-frame color change, so it would erase the heartbeat along with the flicker. We remove that conflict by building each output frame as layers and applying the temporal loss only to the motion and lighting layers, never to the layer that carries the pulse.

Core Claim

Each synthetic frame is the stationary frame, geometrically warped, then multiplied by a generated shading layer and added to a generated artifact layer. Writing it this way has two consequences. First, the known pulse passes through to the output by construction. A warp moves pixels without recoloring them, and the shading layer is penalized for correlating with the specific cardiac waveform. Second, temporal smoothness can be enforced hard on the shading and artifact layers without touching the pulse.

We claim this beats three alternatives on MMPD walking videos for estimators trained on its output. The first is handcrafted motion augmentation. The second is RErPPG-Net-style pulse embedding. The third is direct unsupervised adaptation of an extractor. We also claim that moving the same temporal loss from the layers onto raw pixels measurably suppresses the pulse, which is the conflict described in the research file's Gap 2.

Relation to Prior Work

Three papers supply the closest machinery. Each leaves the walking regime open.

RErPPG-Net treats pulse removal and re-embedding as unpaired translation with two cycles. It cuts UBFC MAE from 44.79 to 0.71 bpm relative to a basic loss, and fine-tuning estimators on its synthetic data reaches 0.41 bpm. It is the strongest prior evidence that synthesized labelled video helps. However, its removal network uses a static background patch as a zero-pulse reference, and it has no inter-frame alignment. Both fail exactly when the subject or camera moves. Our approach never needs to erase a pulse from a moving video. We move a clean, labelled video into the motion domain instead.

Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation supplies the temporal method. It uses a recurrent generator, warping by optical flow (per-pixel motion estimated between frames), and a temporal loss that penalizes disagreement between warped and generated frames. This cut warping error roughly fourfold against CycleGAN (0.000437 vs 0.001748). That loss is applied to whole pixels. On faces it would treat the heartbeat as flicker, so it constrains our design as much as it informs it.

DUNCAN removes MRI motion artifacts without paired data by splitting images into content and artifact codes. It inspires our separation of content from artifact, but it works on static slices with no notion of time.

Two further sources set constraints. The GAN review Generative Adversarial Networks: Recent Developments notes that strict cycle consistency breaks when one domain carries less information. Walking video loses detail relative to stationary video, so we drop the walking→stationary reconstruction requirement rather than force the generator to hide information in faint noise. From radio sensing, the motion discriminator in Remote Respiration Monitoring of Moving Person Using Radio Signals shows that an adversarial term can push motion out of a representation. Our decorrelation term pushes the cardiac waveform out of the motion layers instead.

Research Program

Layered generator. Given a stationary clip $x_{1:T}$ with known contact pulse $p_{1:T}$, we sample a head trajectory $\theta_{1:T}$ from a real walking clip. The trajectory consists of 2D landmark motion and scale, estimated with a standard face tracker. The generator then outputs

$$ y_t = S_t \odot \mathcal{W}_{\theta_t}(x_t) + A_t , $$

where $\mathcal{W}_{\theta_t}$ is a landmark-driven warp, $S_t$ is a positive per-pixel shading map (moving shadows, illumination sweeps), $A_t$ is an additive artifact map (blur residue, specular glints, compression-like texture), and $\odot$ is elementwise multiplication.

The warp is deterministic, so the network learns only $S$ and $A$. These are conditioned on the trajectory and on a lighting code taken from the same walking clip. The pairing between a stationary clip and a walking clip is random, so no paired data is used.

Losses. The generator is trained with four terms:

  1. Realism. A spatio-temporal patch discriminator, in the style of the Temporal-PatchGAN in TSAM, compares $y_{1:T}$ to real MMPD walking clips.
  2. Layer temporal consistency. Flow-warping errors are computed on $S$ and $A$ only, using the known warp rather than estimated optical flow. The flow is exact by construction, so the method avoids the brightness-constancy failure of flow estimators on moving skin.
  3. Pulse decorrelation. Within each facial region, the band-limited mean of $S_t$ and of $A_t$ must have near-zero correlation with $p_{1:T}$. This penalizes the specific cardiac waveform, not the frequency band, so the generator can still produce cadence artifacts at the heart-rate frequency. That overlap is the hard case the estimator must learn.
  4. Identity regularizer. Patterned on DUNCAN's image-quality consistency loss, it keeps $S\approx 1$ and $A\approx 0$ when the sampled trajectory is stationary.

Downstream use. We would synthesize walking versions of the labelled stationary clips from UBFC-rPPG and from MMPD's stationary subset, keeping subjects disjoint from the test subjects. PhysNet and TS-CAN, as implemented in rPPG-Toolbox, are then trained with the original pulse as label.

Baselines

Training-free reference. The do-nothing reference is POS applied directly to MMPD walking video. Its reported walking MAE is 17.05 bpm (MMPD). The naive learned reference is TS-CAN trained only on stationary data, reported at over 25 bpm on walking. We re-run both in rPPG-Toolbox on our exact subject split, so that our splits match the published conditions.

Handcrafted augmentation. This is the most dangerous baseline because it is cheap. It applies the same sampled real head trajectories through the same warp. It then adds parametric shading (random low-frequency gain fields that move with the head) and synthetic blur and JPEG noise. The pipeline and training budget are identical to ours, but no generator is learned. To beat it, we need a lower walking MAE with non-overlapping confidence intervals over subject-level bootstrap.

RErPPG-Net. This is the strongest existing generative competitor. We run it as published: remove the pulse from real MMPD walking clips, embed the stationary subjects' contact pulses, and train the same estimators on the same number of synthetic clips. Its reported results are 0.71 bpm and 0.41 bpm on UBFC. We found no reported result on Lacuna for RErPPG-Net on MMPD walking.

Direct unsupervised adaptation. These are discriminative methods that adapt the estimator to unlabelled walking video rather than generating data. Contrast-Phys reports 0.64 bpm on UBFC. SFDA-rPPG reports 12.38 bpm on COHFACE. We found no reported result on Lacuna for either method on MMPD walking, so we run both on the same unlabelled walking clips our translator sees.

Frame-wise CycleGAN. This baseline translates stationary to walking frame by frame without temporal terms. It tests whether unpaired translation alone suffices.

Ablations. Three ablations isolate the key ingredient:

  1. Moving the temporal loss from the $S,A$ layers to raw pixels $y_t$.
  2. Removing the pulse-decorrelation term.
  3. Replacing the known warp with flow estimated from the generated frames.

Why not just…?

Why not just use handcrafted motion and lighting augmentation? This alternative may work nearly as well. The geometric warp, which is shared by both methods, may account for most of the gain. If it does, the learned shading and artifact layers add little and we would report that. Our bet is that real walking artifacts are not captured by random gain fields: shadows tied to scene geometry, auto-exposure pumping, and compression bursts during motion. The head-to-head comparison above, broken down by MMPD lighting condition, settles the question. Natural light is where we expect the learned layers to matter most.

Why not just adapt an extractor directly on unlabelled walking video, as Contrast-Phys or SFDA-rPPG do? These methods need no generator. But they assume the heart rate is roughly constant over 10–30 s windows and that pulse frequency differs from noise frequency. Walking breaks both assumptions, because cadence sits in the cardiac band. Contrast-Phys itself names extreme activity as open. Synthetic data supplies true labels in exactly the cadence-overlap case. We report MAE separately for windows where cadence and true heart rate are within 5 bpm, where we expect the largest gap.

Why not just re-embed pulses into real walking video with RErPPG-Net? Its realism comes for free, since the video is real. However, its pulse removal relies on a static background reference, and any of the walker's own pulse that survives removal becomes label noise. We measure that leakage directly by running POS on its "pulse-erased" walking clips. Our method never has to erase a pulse.

Why not just deflicker frame-wise translations afterwards? Post-hoc smoothing penalizes high-frequency temporal change and so attenuates the pulse. Ablation 1 above (temporal loss on raw pixels) tests the same mechanism inside our model and quantifies the loss of pulse signal-to-noise ratio.

Proposed Experiments

The primary test is MAE, Pearson correlation, and signal-to-noise ratio on MMPD walking clips from held-out subjects. Secondary tests are MMPD talking and head-rotation clips, PURE's motion setups, and MMSE-HR, where we check that gains do not come at a cost under expression-driven motion. Results are stratified by Fitzpatrick skin type, since TS-CAN's 3.06→15.43 bpm skin-tone gap on MMPD may compound with motion.

To check the synthetic data itself, we use three measurements:

  1. Realism: video FID against real walking clips, plus a walking-versus-synthetic classifier.
  2. Pulse fidelity: POS heart rate on the synthetic clips compared with the known label.
  3. Leakage: the measured correlation between $S,A$ and $p$.

The decisive outcomes are as follows. If the learned translator does not beat handcrafted augmentation on walking MAE, the generative component is unjustified. If moving the temporal loss onto pixels does not reduce pulse fidelity, the layered design is unnecessary. If our estimators beat Contrast-Phys and SFDA-rPPG mainly in cadence-overlap windows, that supports the claim that synthesized labels address a failure that frequency-based self-supervision cannot.