Head Motion as a Teacher: Test-Time Adaptation of Facial Detail Networks Under Domain Shift

Systems that reconstruct a 3D face from a camera usually work in two layers. The first layer is a 3D Morphable Model (3DMM), a statistical face shape built from hundreds of scans, which gets the overall head and expression right but smooths away wrinkles, pores, and creases. The second layer is a neural network that paints that missing fine detail back on as a "displacement map," an image laid over the face surface that says how far each point should be pushed in or out. These detail networks are trained on a few scan collections that skew heavily toward particular populations and studio cameras. When they see a different population, a cheap webcam, compression artifacts, or makeup, they fail without any warning. They either invent wrinkles from noise and pigment or flatten real ones. Test-time adaptation, meaning updating a model on the unlabeled inputs it meets at deployment, is the standard response to this kind of mismatch. But its main tools minimize classification entropy or adjust class centroids, and neither has an obvious meaning for a continuous map of surface offsets.

We propose to adapt the detail network on a short stream of frames of one person, using a signal that comes from the physics of shading rather than from the network's own confidence. As a head moves under fixed lighting, the light falls on each point of the face from a slowly changing angle. Real surface detail is fixed to the face and produces shading that changes predictably with that angle. Pigment such as freckles or makeup is also fixed to the face, but it looks the same under every lighting direction. Sensor noise, compression blocks, and occluding hands are not fixed to the face at all. So the stream itself can say which leftover image signal is geometry, which is color, and which is junk. We use this three-way split to fine-tune a small part of the detail network in seconds, so that its predictions come back to the true surface instead of absorbing the shift.

Core Claim

The decision "is this residual a detail or an outlier?" is exactly what goes wrong for detail networks under domain shift. A short multi-frame stream with natural pose variation provides enough information to make that decision without labels. Adapting the network to this stream-derived signal should beat three things: adaptation that only matches image appearance, adaptation that only recalibrates feature statistics, and simple averaging of per-frame predictions. The gap should be largest when the shift contains pigment-like or noise-like structure that looks like wrinkles.

This sits on a tension already visible in the literature. FOCUS trains a face autoencoder jointly with an outlier segmenter. It treats pixels the 3DMM cannot explain as outliers and uses a "Misfit Prior" precomputed on synthetic data so that hard-to-fit anatomy such as eyebrows is not masked away. Detail networks such as FGNet and HRNet's virtual multi-view displacement regressor exist because real anatomy also falls outside the 3DMM. A static prior cannot know, for an unseen population or camera, which residuals are anatomy. FOCUS motivates our approach because it shows that the unexplained residual must be split. Our approach differs in two ways: the split is estimated per stream at test time, and the geometric part is kept and fed into a detail network rather than discarded.

Research Program

Source model. We train a two-layer detail model in the style of FaceScape. The coarse layer is a fitted 3DMM. The detail layer is an image-to-image network that predicts a static displacement map in face-surface coordinates, plus expression-driven dynamic maps, from the face image unwrapped into those coordinates. Training uses FaceScape scans, whose subjects the authors themselves note are predominantly Asian. This gives a built-in demographic shift when testing elsewhere. For the coarse fit we use an off-the-shelf model, 3DDFA-V3 from the part-segmentation-guided reconstruction work, and keep it frozen. This isolates the detail layer, where the open problem lies.

Test-time signal. Given frames $t=1,\dots,T$ of one person, the coarse fit provides for each frame a head pose, a low-order lighting estimate $L_t$ in the face's own coordinate frame, the unwrapped image $U_t$, and a visibility mask $V_t$. The adapted network $f_\theta$ predicts displacement $D_{\theta,t}$ from each frame, which gives surface normals $n_{\theta,t}$. Jointly with $\theta$ we estimate a per-stream albedo map $a$ (the face's intrinsic color) and per-pixel, per-frame reliability weights $w_t$. The objective is

$$ \mathcal{L}(\theta,a)=\sum_{t,u} w_t(u)\,V_t(u)\,\rho\!\left(U_t(u)-a(u)\,S(n_{\theta,t}(u),L_t)\right)+\lambda_s\sum_t \|D^{\text{static}}_{\theta,t}-\bar D^{\text{static}}_\theta\|^2+\lambda_0\|\theta-\theta_0\|^2 . $$

In this objective, $u$ indexes surface-coordinate pixels, $S$ is the shading predicted from a normal under lighting $L_t$, $\rho$ is a robust penalty, and $\bar D^{\text{static}}_\theta$ is the cross-frame median of the static component. The last term anchors $\theta$ to the source weights $\theta_0$.

The weights $w_t$ are re-estimated in alternation, in an EM-like loop that echoes FOCUS. A residual that no shared albedo and geometry can explain across frames is down-weighted. A residual that is stable on the face but does not change with lighting is absorbed by $a$. Only residuals that are stable on the face and vary with lighting in the way shading should can move the displacement. Following Tent, which adapts only the scale and shift of normalization layers to avoid collapse, we update only normalization parameters and, as a variant, small low-rank adapters, for a fixed budget of a few dozen steps. The target is seconds per stream, against the roughly 30 minutes per object reported for MeTTA.

What we expect, and where it should fail. The separation is a photometric-stereo argument, so it needs lighting directions in the face frame that span enough angles. This is a classical principle, and the contribution here is to use it as a label-free teacher for a learned network under shift. We therefore predict that the gains grow with the range of head rotation in the stream and vanish for nearly frontal, static clips. In those clips the method should fall back to the source prediction, because of the anchor term, rather than degrade. Single images are out of scope. We say so explicitly rather than borrow the synthetic-batch tricks of DUA, which do not supply new lighting directions.

Relation to the closest test-time work. CycleAdapt adapts a body-mesh regressor on video, using a temporal motion denoiser as a source of pseudo-labels. It reports 87.7 mm mean per-joint error on 3DPW at about 74 ms per frame. It adapts pose and global shape only, and its denoiser over-smooths, which is the opposite of what detail needs. MeTTA refines single-view object meshes at test time with a diffusion prior. It has no face structure, no surface-coordinate parameterization, and requires clean masks. IT³ offers a modality-agnostic self-consistency signal. We considered a render-and-repredict variant of it but rejected it. A network trained on clean renderings maps any re-rendered displacement back to itself, so the consistency loss is near zero whether or not the original prediction was correct.

Baselines

The naive reference is the source-trained detail network applied frame by frame with no adaptation, on top of the same frozen coarse fit. Every gain is reported relative to it. We also report the coarse fit alone, to show how much detail the source network adds before and after shift.

Statistics-only adaptation. This baseline applies DUA's momentum-based recomputation of normalization statistics and MemBN's memory queue of statistics to the detail network, over the same frames of each stream. This is the strongest cheap competitor. MemBN in particular stays stable at batch size one, where plain test-time normalization collapses: MemBN reports 12.88% error on CIFAR-10-C against about 90% for plain test-time normalization. We count it as beaten if our method lowers detail-band normal error beyond what MemBN achieves with equal frames and lower or comparable wall-clock time.

Plain appearance-matching adaptation. This is our objective with all $w_t=1$, the albedo fixed to the coarse model's texture, and no static-consistency term. It is the obvious idea at this intersection and the key ablation. If it matches the full method on our "phantom detail" tests, our core claim is wrong.

Fusion without adaptation. Per-frame source predictions are merged into one static map with a visibility-weighted median in surface coordinates. This tests whether adaptation adds anything beyond aggregation.

Direct per-stream optimization. This optimizes a free displacement map and albedo with the identical objective and weights but no network. It tests whether the learned prior matters.

The strongest published coarse-plus-detail systems. On REALY, the part-segmentation paper reports average 3D errors of 1.436 mm for 3DDFA-V3, 1.537 mm for HRN, and 2.010 mm for DECA. These are coarse-dominated, region-aligned errors. We expect our method not to worsen them, and gains should appear mainly in the detail-specific metrics. For detail-band normal error on REALY there is no reported result on Lacuna. We will compute it ourselves for HRN-style and FaceScape-style detail networks.

All adaptive methods see the same frames and the same frozen coarse fit. Each gets a per-method hyperparameter search with an equal number of trials on a held-out split of synthetic streams, and the same cap on optimization steps.

Why not just…?

Why not just recompute normalization statistics on the test stream? Statistics recalibration rescales all feature channels uniformly across the face. REALY shows that reconstruction error is strongly region-specific, with the mouth markedly harder than the nose or cheeks. Recalibration also has no notion of which residual is pigment and which is geometry. On classification, Tent itself reports that loss-driven adaptation beats statistics-only normalization on ImageNet-C (44.0% versus 49.9% error). Still, this is a real threat on pure sensor shifts such as blur or noise without demographic change. Our FaceScape lab-image experiments with synthetic corruptions are designed to show whether it suffices there.

Why not just train robustly offline, as FOCUS does? FOCUS reaches 1.02 mm median error on NoW, against 1.23 mm for Deep3D, without 3D supervision. However, its Misfit Prior is fixed before deployment, and it discards unmodelable residual rather than recovering detail from it. Our albedo-edit experiment, described below, tests the difference directly. We will also run a FOCUS-style masked appearance loss, with its static prior, as an adaptation variant.

Why not just train on more diverse or augmented data? Augmentation can cover sensor corruptions that someone thought to simulate. It cannot supply scans of under-represented faces that were never captured, and no 3D detail dataset is known to be demographically balanced. Theory on domain adaptation bounds target error by source error plus a divergence term that training on the source alone does not shrink. We will include a source model trained with the full corruption augmentation suite. If that model closes the gap on held-out corruption types, the sensor-shift half of our claim fails.

Why not just optimize the displacement map directly per stream, with no network? This is the most serious alternative, because the objective alone might do all the work. Our position is that the network supplies a prior where the shading cue is weak (small rotations, occluded or self-shadowed regions) and supplies dynamic wrinkles that a static map cannot represent. The direct-optimization baseline above, swept over pose range, settles the question. If it matches our method across pose ranges, we will report that the contribution is the separation rule and not the adaptation.

Proposed Experiments

Data. We build controlled streams by rendering REALY's 100 LYHM scans and held-out FaceScape scans along short head-rotation trajectories under fixed environment lighting. The rotation range is varied from near-frontal to wide. We add corruption families modeled on ImageNet-C (noise, blur, compression, low light) at graded severities. REALY's own five synthesized views per scan serve as an uncontrolled five-frame stream. FaceScape lab multi-view images provide sensor shift without demographic shift, while REALY provides demographic shift. Metrics are REALY's region-aware errors for nose, mouth, forehead, and cheek. We add a detail-band normal error, computed after subtracting a low-pass version of both the scan and the prediction, and FaceScape's point-to-surface distance, which its authors report at 1.22 mm in-domain.

The decisive check: phantom detail. We edit only the texture of test scans by painting dark wrinkle-like lines, freckles, and makeup, so the true geometry is unchanged. We also overlay occluders and noise that are not attached to the face. We measure displacement magnitude in the edited regions, which should be zero. Plain appearance-matching adaptation should convert painted lines into grooves. The full method should push them into albedo. The converse test uses scans with real deep wrinkles rendered with flat albedo: here the method must recover the grooves rather than explain them away as color. If our method fails either test, the separation claim is false.

Further checks. We plot the gain over source as a function of head-rotation range, to confirm that the gain vanishes near frontal and that performance falls back to source rather than degrading. We report gains per region and per demographic subgroup in REALY, to check that adaptation helps the populations the source data under-represents. We run an adaptation-scope ablation comparing normalization-only updates, low-rank adapters, and full fine-tuning, and record when collapse appears. Finally, we report wall-clock time per stream against MeTTA's reported runtime, so that speed claims rest on measured numbers.