When someone wears a virtual reality headset, more than half of their face is hidden. Telepresence systems fill in the hidden region with a generative network that paints a plausible face, conditioned on a selfie taken earlier and on whatever skin stays visible. These systems are judged on whether the output looks real and keeps the person's identity. Nobody checks whether the painted skin behaves like living tissue. Real facial skin changes color very slightly with every heartbeat, and a camera can recover that signal, a technique called remote photoplethysmography (rPPG). A completed face can fail at this in two ways. It can invent a heartbeat-like flicker that has nothing to do with the wearer's actual pulse. It can also wash out the real pulse in the visible lower face during blending. Both failures are a form of anatomical hallucination: the network fabricates or erases a biological feature that the observations do not support.
This proposal makes the pulse a measurement tool. The plan is to hide parts of faces artificially in public rPPG videos, where the "hidden" skin was in fact filmed and its true pulse is known. That lets us score, region by region, whether a completion network's output reproduces, invents, or erases blood-flow signal. Head motion, and headset slippage in real use, can masquerade as pulse, so the score is built to discount it. We then test whether a cheap physiological-consistency term, used during training or applied after the fact, removes the failures without hurting appearance. The claim is new in two respects. Physiological fidelity becomes a measurable property of facial completion, and segmentation into skin regions is what makes that property both measurable and correctable.
Core Claim
Reference-conditioned face completion for headset wearers produces pulse-inconsistent skin. This happens both inside the painted region and in the visible region it blends into. Existing quality measures, which focus on appearance and identity, do not register the problem. A region-wise, motion-discounted perfusion-consistency score detects it. The score is computed against ground truth that only becomes available when occlusion is simulated on existing rPPG video.
The three source directions play different roles in this claim. Headset occlusion is the application setting. Anatomical hallucination is the failure we measure. Region-of-interest segmentation is the method: skin is split into anatomical patches so that the pulse can be compared locally. Motion artifacts are the main confound the measurement has to survive.
Why This Gap Exists
Face completion for headsets has advanced steadily. Mask-off and Identity Preserving Face Completion for Large Ocular Region Occlusion condition generative inpainting on unoccluded reference photos. Attention based Occlusion Removal for Hybrid Telepresence Systems and Show Your Face use attention to align reference identity with live partial views. All of them optimize for visual plausibility. The research file notes that this risks "synthetic expressions uncoupled from real emotional states." None of them evaluates temporal physiological content.
On the medical-imaging side, Rethinking Perceptual Metrics for Medical Image Translation shows that Fréchet Inception Distance (FID), a standard image-realism score, has near-zero correlation with anatomical fidelity. That paper motivates our premise that realism scores reward hallucination. Its proposed remedy, a downstream segmentation Dice score, is static, so it cannot see a fabricated time signal. TPOT supplies the method template we reuse: a frozen auxiliary network extracts a structure (there, vessel topology), and a loss penalizes changes to it. We replace the static vessel topology with a temporal pulse structure.
The perfusion literature supplies the measurement tools. Perfusion assessment via local rPPG computes per-region perfusion index and correlation with a reference tissue. With those features it detects silicone and rigid spoofing masks at 96.8% accuracy, which shows that local pulse cleanly separates living from non-living surfaces. It only flags occluders, though; it never asks whether synthesized skin is physiologically honest. Contrast-Phys shows that pulse is spatially redundant across the face, a property we exploit. It also assumes noise is spatially uncorrelated, and headset slippage breaks that assumption, which constrains our design. The closest cross-direction work, Emotion Recognition in HMDs, fuses the half-visible face with contact EEG and skin-conductance sensors. It neither reconstructs the face nor uses camera-based pulse, and its authors report that it is sensitive to head motion.
Research Program
The first component is a testbed with ground truth. We take UBFC-rPPG, PURE, and MMPD, each of which pairs facial video with a contact pulse oximeter. We apply headset-shaped occlusion masks, following the artificial-occlusion procedure of Pei et al., and add rigid jitter of the mask to mimic slippage. Because the original frames are unoccluded, the true local pulse in the forehead and periocular patches is known. PURE's scripted head motions and MMPD's stationary, rotation, talking, and walking regimes give graded motion stress. MMPD also spans Fitzpatrick skin types 3 to 6, which matters because the pulse signal weakens as melanin increases.
The second component is the score. We segment skin into anatomical patches using landmarks, following the region design of the perfusion work. In each patch $r$ we extract a pulse trace with a classical estimator. For each patch we compute the spectral coherence $C(x_r, p)$ between the completed video's trace $x_r$ and a reference pulse $p$. In simulation, $p$ is the oximeter signal, or the trace from the original unoccluded patch. In deployment, $p$ is the trace from the observed lower face. We also compute the coherence between $x_r$ and a head-motion trace $m$ taken from landmark tracking. The per-patch fidelity is
$$F_r = C(x_r, p) - \max\big(0,\; C(x_r, m) - C(p, m)\big),$$
which subtracts only the share of agreement that can be explained by motion beyond what the true pulse itself shares with motion. We report three quantities separately: $F_r$ inside the painted region, $F_r$ in the visible region (to catch erasure), and pulse amplitude relative to the original (to catch fabricated or suppressed amplitude).
The third component is the fix, under two variants. In the training-time variant, we add a frequency-domain consistency loss. A frozen deep rPPG network reads patch traces from the completed output. The loss asks their power spectra to match the reference from observed skin, in the style of Contrast-Phys, which tolerates phase delays between regions. It also asks visible-region traces to match the input. In the post-hoc variant, we leave the generator untouched. We modulate the painted skin's chrominance with the observed lower-face pulse, scaled by a per-region amplitude learned from the unoccluded training videos.
A known risk, flagged in the file as domain shift in guidance networks, is that the network learns to fool the frozen pulse extractor. To guard against this, the score always uses a different estimator, POS, than the one used in training. We also measure, separately, how much the training-time extractor and POS disagree on generated video.
Baselines
The naive reference point has two parts. The first is no completion: a flat mean-skin fill of the occluded region, which by construction carries no pulse. The second is a static warp of the neutral reference selfie, which carries no pulse in the painted region but leaves visible pixels untouched. These set the floor for $F_r$ inside the painted region and the ceiling for visible-region preservation.
The strongest existing competitors are the reference-conditioned completion networks: Show Your Face, Attention based Occlusion Removal, and Identity Preserving Face Completion. They are the systems that would actually be deployed, and attention-based alignment to live patches is the mechanism most likely to transfer real pulse incidentally. Where code is unavailable, we reimplement them on one shared generator backbone. Every model is trained on the same occluded training split, with the same reference frames, the same number of GPU hours, and the same hyperparameter search budget. Reported numbers come from the held-out subjects. No reported result found on Lacuna for any pulse-fidelity measure of face completion, so the comparison is new by construction.
For the downstream heart-rate task, the comparison point is rPPG on the visible lower face alone, using Contrast-Phys and the supervised models in rPPG-Toolbox. On unoccluded UBFC-rPPG, Contrast-Phys reports a mean absolute error (MAE) of 0.64 bpm. On walking video in MMPD, the strongest models exceed 25 bpm MAE (MMPD). No reported result found on Lacuna for heart rate under headset-style occlusion.
Beating the competitors means two things. Painted-region $F_r$ must rise by a margin larger than the across-seed spread. Visible-region $F_r$ must not drop. Identity similarity and the Sliced Wasserstein Distance favored by Konz et al. must stay within noise of the baseline generator.
Three ablations isolate the key ingredients. Removing the motion term from $F_r$ tests whether motion discounting changes model rankings under head rotation and walking. Replacing patch-wise segmentation with a single whole-occluded-region trace tests whether the region-of-interest structure matters. Swapping the frequency-domain loss for a time-domain correlation loss tests the phase-invariance choice.
Why not just…?
Why not just estimate heart rate from the visible lower face and skip completion entirely? For the narrow goal of estimating heart rate, this is likely as good or better, and we will show it. This proposal does not claim better heart rate. The question it asks is whether the rendered avatar that other people see, and that downstream affect models consume, carries fabricated or erased physiology. Lower-face rPPG serves as the yardstick: a completion whose visible-region pulse is worse than the raw lower face's is erasing real signal.
Why not just use FID, identity similarity, or landmark error to judge completions? Konz et al. found FID nearly uncorrelated with anatomical fidelity in medical translation. All of these metrics are also computed frame by frame, so they cannot register a temporal signal by design. We test this directly. Across all baseline generators and checkpoints, we compute the rank correlation of each standard metric with ground-truth painted-region pulse fidelity. If the standard metrics track pulse fidelity well, the score adds nothing, and we will say so.
Why not just transplant the observed pulse onto the painted skin as a post-processing step? This is a real threat, which is why it is one of our two fix variants rather than a straw man. If post-hoc modulation matches the training-time loss on $F_r$ and costs nothing in appearance, the right recommendation is the cheap fix. The contribution then shrinks to the testbed, the score, and the audit showing that current systems fail. The training-time variant can still win in one plausible case: blending in the visible region erases pulse, which a post-hoc pass over the painted region cannot repair.
Why not just use the spoof-mask perfusion classifier of Kossack et al.? Their classifier answers a yes-or-no question, live surface or not. It is not graded against a true pulse, and it does not discount correlated motion. We run their features with the same SVM as an additional detector. The test is whether it separates honest from fabricated completions as well as $F_r$ does on the simulated ground truth.
Decisive Experiments
The first check is the audit. On simulated occlusion across UBFC-rPPG, PURE, and MMPD, do existing completion networks show painted-region $F_r$ near the flat-fill floor, or spurious amplitude, or reduced visible-region $F_r$ relative to the raw input? If none of these failures appears, the central premise is false and the project stops there.
The second check is the metric's validity. Does $F_r$ rank models in the same order as their true painted-region pulse error against the original video? Does that ranking hold across the MMPD motion regimes and skin types? Does removing the motion term reorder models under walking?
The third check is the fix. Does either variant raise painted-region $F_r$ without visible-region loss or appearance cost? Does the gain survive when scored with POS rather than the training extractor? Does it still hold on a small real-headset capture? That capture would pair an external lower-face camera with a finger oximeter, in the style of the Pei et al. setup, where only the reference-from-observed-skin version of the score is available.
References
- Mask-off: Synthesizing Face Images in the Presence of Head-mounted Displays
- Identity Preserving Face Completion for Large Ocular Region Occlusion
- Attention based Occlusion Removal for Hybrid Telepresence Systems
- Show Your Face: Restoring Complete Facial Images from Partial Observations for VR Meeting
- Rethinking Perceptual Metrics for Medical Image Translation
- TPOT: Topology Preserving Optimal Transport in Retinal Fundus Image Enhancement
- Perfusion assessment via local remote photoplethysmography (rPPG)
- Contrast-Phys: Unsupervised Video-Based Remote Physiological Measurement via Spatiotemporal Contrast
- Emotion Recognition in HMDs: A Multi-task Approach Using Physiological Signals and Occluded Faces
- rPPG-Toolbox: Deep Remote PPG Toolbox
- MMPD: Multi-Domain Mobile Video Physiology Dataset