Skin-Anchored Gaussians for Arbitrary-Scale Video Face Super-Resolution

When a face video is enlarged, the fine detail that the enlargement model invents (pores, stubble, fine lines) should stay attached to the skin as the person talks and turns. Current systems fail at this in one of two ways. Image models that enlarge by any factor, including non-integer zooms such as ×3.7, process each frame independently, so the invented detail flickers from frame to frame. Video models that do enforce consistency across frames either work only at a fixed ×4 or align frames with optical flow, a per-pixel motion estimate between neighbouring frames. Flow is least reliable on exactly the parts of a face that matter: smooth cheeks, a moving mouth, and heads turning to profile. The result is "texture sliding," where the skin pattern appears to float over the face.

We propose to store the invented detail on the face surface itself rather than in image pixels. The high-resolution face is drawn as a set of small 2D Gaussian blobs, the fast any-scale representation introduced for still images. Each blob, however, is permanently attached to a point in a dense canonical face coordinate system, such as a UV map in which every skin point has a fixed address regardless of pose or expression. In each frame, a dense face-correspondence network places the blobs in the image. The blobs' colour and detail are refined over time by a learned filter that updates each blob's own state. Two things follow from this design. Enlargement to any factor comes from rendering the blobs at any resolution. Consistency across frames comes from the fact that the same blob represents the same patch of skin in every frame. Frames are aligned by looking each one up against the fixed face coordinates, not by chaining flow estimates, so alignment errors should not pile up over long clips.

Core Claim

Temporal consistency in arbitrary-scale face video super-resolution is best obtained by anchoring the continuous representation's primitives to a dense canonical face surface, not by warping image-space features between frames. We predict three measurable consequences, each checked against a baseline in the experiments below. First, texture sliding and identity flicker drop more on weakly textured regions (cheeks, forehead) than on landmark-rich ones. Second, the benefit grows with clip length, whereas flow-propagated methods plateau. Third, quality holds at enlargement factors never seen in training, because rendering Gaussians at a new resolution requires no new learned upsampling filter.

The idea sits at a gap that each of the closest works leaves open.

ARASFSR showed that any-scale implicit decoders (small networks that map a pixel coordinate to a colour) distort faces unless given a global position prior. Its prior is a 2D positional code on pre-aligned stills, and the model has no temporal mechanism. Our anchoring replaces that 2D code with a surface address that moves with the face.

ST-AVSR and DCGU do combine arbitrary-scale upsampling with temporal context. Both align frames with flow or patch search and carry no face prior, and DCGU's hashing-based patch search is the kind of matching that struggles on smooth skin.

KEEP propagates face-restoration state across frames with a Kalman-style filter, which directly reduces identity and landmark jitter. It is fixed to ×4, its state lives in CodeFormer's quantised codebook, and its prediction step uses flow.

We reuse two things from prior work. The Gaussian rendering and position-drift ideas come from ContinuousSR, which reports rendering any scale in 1 ms. The adaptive-gain update comes from KEEP. What we change is where the state lives: on the surface, not in the image.

Research Program

The system has four parts.

Correspondence. A dense correspondence network maps every low-resolution face pixel to a canonical surface coordinate. The default is PRNet's UV position map, which runs at 9.8 ms per frame and stays stable under 90-degree rotations. Densemarks is the variant: it also covers hair and neck and roughly halves same-person correspondence error relative to DINOv3 (MAE 3.68 vs 7.60), but needs about 16 GB just for its feature grid. We run the correspondence network on the low-resolution input, and it is fine-tuned jointly on degraded inputs, since both correspondence networks were trained on clean images.

Canonical Gaussian bank. A fixed bank of Gaussians is defined in canonical UV space. Their density follows the canonical layout, and their positions may drift locally, as in ContinuousSR, but only within UV space.

Recurrent state update. Each Gaussian $k$ carries an appearance state $a_k$ (colour plus a high-frequency code). At frame $t$ the state is updated as

$$a_k^{t} = a_k^{t-1} + g_k^{t}\,\big(o_k^{t} - a_k^{t-1}\big),$$

where $o_k^{t}$ is the appearance evidence gathered from frame $t$'s low-resolution features at the Gaussian's projected location, and $g_k^{t}\in[0,1]$ is a learned per-Gaussian gain. The gain is predicted from the Gaussian's visibility, how obliquely the surface faces the camera, and how much $o_k^{t}$ disagrees with the stored state. Occluded Gaussians receive $g=0$ and keep their memory, which is the mechanism for surviving head turns.

Rendering. The frame is rendered at the requested scale by splatting the Gaussians with their current image positions and covariances.

Two further components handle what the surface model cannot. Accumulating state would erase transient detail such as expression wrinkles, lip contact and eye glints. A lightweight image-space residual branch therefore predicts frame-specific high-frequency detail on top of the rendered result, and a regulariser penalises the residual in proportion to how well it is explained by the canonical state. Non-face regions (background, and hair and shoulders in the PRNet variant) are handled by a standard image-space any-scale branch and blended using the correspondence network's visibility mask.

Training. Training uses VFHQ clips degraded at scales sampled continuously in ×2–×4. Out-of-range scales up to ×8 are held out. The losses are reconstruction and perceptual losses, plus a surface-consistency loss: points tracked with CoTracker3 on the ground-truth high-resolution clip should carry the same rendered texture across frames.

Baselines

The naive reference renders the Gaussians from ContinuousSR applied independently to each frame. It is trained on the same VFHQ data and scale range, with no temporal state and no anchoring. This isolates how much flicker the per-frame any-scale approach produces. A second reference is bicubic enlargement.

The strongest existing competitor for arbitrary-scale video is ST-AVSR. It is the only cited method that unites a scale-aware upsampler with both past and future temporal context, and it reports about a +1.5 dB PSNR gain on REDS (ST-AVSR). DCGU reports 32.57 dB PSNR on REDS4 at ×4 (DCGU). Neither was evaluated on faces, and there is no reported result found on Lacuna for any arbitrary-scale video method on VFHQ. We will therefore retrain both from their published recipes on the identical VFHQ split, degradation pipeline, scale sampling, iteration budget and hyperparameter search budget. We will also fine-tune a version of each from its REDS checkpoint, so that neither is disadvantaged by initialisation.

For fixed-scale face video, KEEP is the strongest competitor on consistency metrics, reporting a +0.8 dB PSNR gain on VFHQ-mild over the next-best method (KEEP). We compare against it at ×4, its native setting, and at other scales via ×4 output plus resizing.

Beating the field means two things. At ×4 we must at least match KEEP on PSNR and LPIPS. At every scale we must lower flicker, identity fluctuation and texture sliding relative to face-retrained ST-AVSR, with the gap widening at held-out scales and on longer clips.

The key ablation keeps everything identical but removes anchoring: the Gaussian states are propagated in image space with RAFT flow instead of through canonical UV lookup. If this flow-propagated twin matches the full model, the surface anchoring is not what matters. A second ablation removes the residual branch, to measure how much transient expression detail the canonical memory alone destroys.

Why not just…?

Why not just run KEEP at ×4 and resize to the target scale? ARASFSR reports that resizing fixed-scale outputs blurs pores and eyelashes and distorts structure at unseen scales, reaching 30.97 dB at ×8 on CelebA-HQ against the 30.60 dB of the DIINN implicit baseline. However, KEEP's temporal machinery is strong, and at scales near ×4 this cascade may be hard to beat. We include it at ×2.5, ×4, ×6 and ×8. We expect to lose or tie near ×4 on fidelity and to win clearly at ×6 and above, and we will report the result whichever way it falls.

Why not just enlarge each frame with ContinuousSR and smooth afterwards? The Deep Video Prior smoother (DVP) needs about 2 s per frame of test-time training and must be stopped at the right point to avoid either blur or returning flicker. A cheaper exponential moving average in image space averages detail that has slid, instead of detail that belongs to the same patch of skin. We run both, and we measure whether they suppress flicker only by blurring, which shows up as lower LPIPS quality and loss of high-frequency energy on cheeks.

Why not just fine-tune ST-AVSR on faces, or feed it dense landmarks as an extra input? This is the real threat. A capable flow-based video model trained on enough face video might learn to handle skin. The face-retrained ST-AVSR baseline tests the first half of the question. We also train a variant that receives the PRNet UV map as an extra input channel while still aligning frames with flow. If that variant closes the gap, the conclusion is that dense geometry helps as conditioning and that anchoring the representation is unnecessary. We would then report the narrower claim.

Why not just smooth 68-point landmark tracks and condition on them? Sparse points give no constraint over the cheeks and forehead, where sliding occurs (Sparse to Dense Motion Transfer found sparse driving misses mouth and gaze micro-motion). The region-stratified metrics below test this directly: a sparse-conditioned variant should improve near landmarks but not on smooth skin.

Proposed Experiments

The main benchmark is VFHQ, degraded at continuous scales ×2–×8 including non-integer ones. VoxCeleb is used for talking-heavy clips (MAFN reports 34.81 dB PSNR there at fixed ×4 with 7 frames (MAFN), which we reproduce as a sanity check). CelebV-HQ provides clips with large pose variation.

Fidelity is measured with PSNR, SSIM and LPIPS. Consistency is measured with identity fluctuation $\sigma_{IDS}$ and landmark jitter $\sigma_{AKD}$ as in KEEP, plus a flow-free flicker measure. Texture sliding is measured by tracking points with CoTracker3 on the ground truth and comparing the local texture patches of the output along those tracks. All metrics are stratified into landmark-rich and smooth-skin regions using face parsing.

Four checks decide the claim. First, the smooth-skin sliding error must fall relative to the flow-propagated ablation, by more than the landmark-rich error falls. Second, the gap between our model and flow-based baselines must grow with clip length from 5 to 50 frames; MAFN's observation that flow warping stops helping beyond about 5 frames predicts that the baselines will plateau there. Third, the advantage must hold or grow at the held-out ×6–×8 scales. Fourth, on sequences with yaw beyond 60 degrees, identity fluctuation after the face turns back to frontal must stay lower than KEEP's, which tests whether frozen occluded states actually preserve memory.

If the full model does not beat both the ST-AVSR variant given the UV map as extra input and the flow-propagated ablation on the smooth-skin metrics, the anchoring hypothesis is falsified.