A face video carries a faint record of the heartbeat. Each pulse slightly changes skin color, and a camera can pick this up as remote photoplethysmography (rPPG), meaning pulse sensing from video without touching the skin. Recent work shows that the fine shape of this pulse can identify a person with near-zero error. That result holds only when the person keeps still and stays in one physiological state. Head motion shifts which patch of skin each pixel sees, and exercise or stress reshapes the pulse itself. Current rPPG biometrics break under both. We propose that the two failures share a single fix. Medical dynamic imaging learned this fix the hard way: estimate tissue motion and the physiological signal together, under a physical model of the signal, instead of registering first and fitting afterwards.
Concretely, we would track the face onto a fixed anatomical surface. We would then model the pulse at every skin location as one shared heartbeat waveform passed through a short, location-specific filter, which we call the local transfer function. Motion estimation and this pulse model are solved jointly, so color changes caused by blood are not mistaken for movement. The identity template is the map of transfer functions relative to a reference site, with the shared waveform divided out. Our central bet is that heart-rate and blood-pressure changes mostly alter the shared waveform. The way the pulse is delayed and filtered on its way to the cheek versus the forehead should mostly reflect a person's facial vascular anatomy. If that holds, a template built from relative transfer functions should be both motion-tolerant and more state-stable than the absolute waveform morphology used today.
Core Claim
A jointly registered, physically parameterized "pulse transfer map" will give lower equal error rate (EER, the operating point where false accepts equal false rejects) than morphology-based rPPG biometrics in two conditions. The first is head motion. The second is enrollment at rest followed by verification after exercise. The gain should come from two identifiable ingredients: joint rather than decoupled motion estimation, and relative rather than absolute pulse features. Each ingredient gets its own ablation.
Where This Sits
Two lines of prior work motivate the proposal. Biometric Authentication Based on Enhanced Remote Photoplethysmography Signal Morphology shows that rPPG pulse shape is a strong identifier. It reports intra-session EER of 0–0.17% and post-exercise EER of 2.16% on OBF. It deliberately scrambles pixels into a de-identified "color soup", so it discards all spatial correspondence. The authors report that performance degrades significantly under head motion on PURE, and they do not test multi-day sessions.
Impairing Factors in Remote-PPG Pulse Transit Time Measurements on the Face plays two roles for us. It measures facial rPPG against a carotid reference and finds that pulse shape distorts differently across facial arterial territories. That is evidence that spatial pulse structure is site-specific, which is exactly what we want to exploit as identity. The same paper also constrains the design. It finds ±30° limits of agreement in phase, and it shows that during handgrip exercise, shape changes were misread as timing shifts. We take that failure as the reason to model filters rather than scalar delays. We also treat it as the main risk: relative filters may still drift with state. Our experiments test this directly rather than assuming it away.
Two further works come closest on the motion side, and neither targets identity. Mask Attack Detection Using Vascular-weighted Motion-robust rPPG Signals tracks facial patches through pose changes using feature-point matching and dynamic programming. It weights regions with a facial arterial atlas and reaches 0.87% EER on 3DMAD. However, it only decides live-versus-mask, and it tracks first and analyzes afterwards. Motion Matters: Neural Motion Transfer for Better Camera Physiological Measurement makes rPPG heart-rate estimators robust by synthesizing moving training videos. It evaluates only average heart rate and never models tissue correspondence. We reuse its motion-transfer generator as a controlled test harness, not as the method.
Three medical-imaging works supply the method. In Optimal Estimation of Diffusion in DW-MRI by High-Order MRF-Based Joint Deformable Registration and Diffusion Modeling, registering images to a physical signal model beats register-then-fit when contrast changes over time. Robust and Efficient Alignment of Calcium Imaging Data through Simultaneous Low Rank and Sparse Decomposition shows that joint alignment with a low-rank background can be solved by plain gradient descent, 100–1000× faster than decompositions based on the singular value decomposition. We copy that optimization style, because the graphical-model solvers are too slow for authentication. TAI-GAN shows that motion correction degrades when intensity change is physiological. Its fix is to translate frames into a common contrast state. We instead model the physiological change explicitly, which avoids needing segmentation masks.
On the physiology side, DE-PADA supplies the structural analogy. For ECG, it splits each heartbeat into a heart-rate-stable part and a heart-rate-sensitive part. Our shared-waveform/relative-filter split is the spatial counterpart for optical pulse data. DE-PADA works on contact sensors with no spatial dimension, and its post-exercise identification rate still stops at 68.95%. Seeing Red shows how severe cross-day drift is for single-site pulse morphology: EER rises from about 8% within a session to 20% across sessions. This motivates the move away from absolute single-site shape.
Method
A pretrained 3D face-mesh fit maps each frame to canonical surface coordinates $u$. A residual non-rigid deformation $d_t$ at frame $t$ then absorbs mesh-fitting jitter and skin sliding. Skin color at surface point $u$ is modeled as a slowly varying low-rank term $L(u,t)$, which captures shading, illumination and slow thermal drift. On top of that sits a pulsatile term, the shared driving pulse $p(t)$ convolved with a short local filter $h_u$ of a few taps. The joint objective is:
$$ \min_{d,\,L,\,p,\,h}\; \sum_{t,u} \big\| I_t\big(W_{d_t}(u)\big) - L(u,t) - (h_u * p)(t) \big\|^2 + \lambda_d R(d) + \lambda_h \sum_{u\sim v}\|h_u - h_v\|^2 \quad \text{s.t. } \mathrm{rank}(L)\le r $$
The terms are defined as follows. $I_t$ is the frame, and $W_{d_t}$ is the mesh warp plus residual deformation. $R$ is a deformation smoothness penalty. The last term, summed over neighboring surface points $u \sim v$, encourages filters to vary smoothly across the face.
Because the pulse term explains the periodic color change, the motion variables cannot absorb it, and vice versa. This is the lesson carried over from joint registration in diffusion MRI. The problem is solved by alternating gradient steps with a linear projection for $L$, following the fast approach of the calcium-imaging work above.
The template is built in three steps. First, each $h_u$ is expressed relative to a reference-region filter $h_{\text{ref}}$, as a frequency-domain ratio restricted to the first few cardiac harmonics. Second, these ratios are resampled on a fixed coarse grid of the canonical mesh. Third, a small metric-learning network embeds the result. The template contains no facial appearance, only blood-propagation parameters, so it keeps the privacy spirit of the color-soup approach while restoring spatial correspondence.
Research Program
The work has three parts. The first is building the joint estimator and validating that it recovers known motion and known per-site delays in controlled synthetic video. For this we inject driving-video motion via the Paruchuri et al. generator into still clips, and we insert known spatially varying pulse delays. The second is a stability study. For each subject with rest and post-exercise recordings, we compare within-subject drift and between-subject separation for absolute morphology templates versus relative transfer templates. The third is the biometric evaluation proper, stratified by motion magnitude, physiological state and skin tone.
Baselines
The naive reference is a fixed-rectangle facial region with no motion handling. From it we extract a single averaged pulse, compute fiducial morphology features in the style of Seeing Red, and match them with a distance-based template. This shows how much any spatial modeling helps at all.
The strongest existing method is the rPPG morphology authentication of Sun et al. (paper), retrained with its contrastive pre-training and contact-PPG supervision on our exact subject splits. Its reported numbers are intra-session EER of 0–0.17% and post-exercise EER of 2.16% on OBF. For PURE under head motion, only a qualitative statement of severe degradation exists; no reported result found on Lacuna for its PURE EER. Beating it means matching it within noise on still, rest data, and producing a clearly lower EER on the PURE and AFRL motion subsets and on OBF rest-to-post-exercise verification. We will report confidence intervals from subject-level bootstrap.
Two engineered variants of Sun et al. are the strongest simple competitors. One feeds it landmark-stabilized crops. The other trains it on motion-augmented videos from the Paruchuri generator. Both get the same training subjects, the same number of hyperparameter trials, and comparable GPU hours.
Two ablations isolate the key ingredients. The decoupled ablation first registers frames using the same mesh-plus-flow machinery, then fits the pulse model; it tests whether joint estimation matters. The absolute-template ablation keeps joint estimation but uses $h_u$ without dividing by the reference; it tests whether the relative representation is what buys state stability.
Face recognition on the same videos is reported as an upper-reference point. We add a control that trains our embedding on canonical mesh geometry alone. If geometry alone matches our EER, the template is leaking face shape rather than encoding blood flow.
Why not just…?
Why not just stabilize the face with a landmark tracker and keep the existing morphology model? Yao et al. report that landmark detectors jitter at sub-pixel scales that destroy rPPG signals, which is why they resort to costly keypoint matching (paper). Even with perfect tracking, the model still uses absolute single-signal morphology, which Seeing Red shows drifts to 20% EER across days. The landmark-stabilized baseline tests the first half of this argument directly. If it closes the motion gap, the joint estimator is unnecessary for motion, and the contribution narrows to the relative template.
Why not just train on motion-augmented video? Motion transfer improved heart-rate error by 47% cross-dataset on PURE. However, the authors themselves flag generative warping artifacts under large rotations, and they never verified that fine morphology survives. Augmentation also does nothing about exercise-induced drift. This is a real threat on the motion axis, which is why it is a full baseline. The synthetic-injection study measures whether augmentation preserves per-site pulse detail.
Why not just apply DE-PADA-style domain adaptation to rPPG features? Domain adaptation needs multi-state auxiliary data and still left ECG at 68.95% post-exercise identification. It also does not address spatial misalignment. We will add a domain-adaptation head to the Sun et al. baseline on OBF, using held-out subjects as the auxiliary cohort. If this matches the relative template on state shift, the physical factorization is not earning its keep.
Why not just use face recognition? On raw video it will likely win on accuracy. The target setting is different: a template that stores no appearance, that proves live blood flow, and that cannot be satisfied by a mask or a photograph. The geometry-only control checks that our template is not face recognition in disguise.
Proposed Experiments
The datasets are OBF (100 subjects, rest and post-exercise) for state shift and skin-tone stratification, PURE and AFRL for controlled head motion, UBFC-rPPG for still baselines, UBFC-Phys for stress tasks, and MMPD for mobile capture across skin tones. None of these offers multi-week sessions. We will therefore collect a small longitudinal set with a consumer camera and contact PPG across several weeks, and we label it a secondary result.
Metrics are EER and AUC for verification. For the stability analysis we report the ratio of within-subject template drift across states to between-subject distance. For the synthetic harness we report recovery error of injected delays and motion.
Four checks decide the outcome. First, on PURE and AFRL, the joint method must beat the decoupled ablation. If it does not, joint estimation is unjustified. Second, on OBF rest-to-post-exercise, the relative template must show smaller drift ratio and lower EER than the absolute template. If it does not, the Moco et al. finding that exercise reshapes local pulses extends to relative filters too, and the central bet fails. Third, the geometry-only control must perform far worse than the full template. Fourth, the gap between light- and dark-skinned groups must be reported against Sun et al.'s 2.52% versus 4.04%. Pooling many sites may help darker skin through averaging, but we do not assume it.
References
- Biometric Authentication Based on Enhanced Remote Photoplethysmography Signal Morphology
- Impairing Factors in Remote-PPG Pulse Transit Time Measurements on the Face
- Mask Attack Detection Using Vascular-Weighted Motion-Robust rPPG Signals
- Motion Matters: Neural Motion Transfer for Better Camera Physiological Measurement
- Optimal Estimation of Diffusion in DW-MRI by High-Order MRF-Based Joint Deformable Registration and Diffusion Modeling
- Robust and Efficient Alignment of Calcium Imaging Data through Simultaneous Low Rank and Sparse Decomposition
- TAI-GAN: A Temporally and Anatomically Informed Generative Adversarial Network for early-to-late frame conversion in dynamic cardiac PET inter-frame motion correction
- DE-PADA: Personalized Augmentation and Domain Adaptation for ECG Biometrics Across Physiological States
- Seeing Red: PPG Biometrics Using Smartphone Cameras