Camera-Free Identity Tokens for Feed-Forward Head Avatars from Uncalibrated Monocular Video

A phone video of someone's face hides a basic confusion. A head that moves toward the lens, a lens with a wider field of view, and a jaw that juts forward can all produce nearly the same change in the image. Current fast avatar systems, which turn images into a 3D head in about a second, avoid this confusion by assuming calibrated cameras. The monocular systems that accept casual video instead fit each person with ten minutes to several days of optimization. In both cases the camera's focal length and distance are never cleanly separated from the person's face shape. Whatever the model fails to attribute to the camera ends up in the face. It shows up either as a distorted skull, which is identity error, or as fake expression motion when the person merely leans in.

We propose a single design principle for a feed-forward head-avatar model, meaning one trained network that produces the avatar in one pass with no per-person optimization. The parts of the network that build the person's identity and expression never see camera information. Everything that depends on the camera (focal length, distance, head pose) is estimated by a separate small module and applied only at rendering time. One argument makes this workable on uncalibrated video. Under perspective projection, the leftover scale ambiguity for a rigid head reduces to one number per video, and that number is constant over time, so it cannot leak into per-frame expression. The focal length itself can be recovered from how perspective distortion changes as the head rotates. The same principle, keeping camera conditioning out of the identity representation, is what Condition Matters in Full-head 3D GANs found suppresses multi-face ("Janus") artifacts in 3D generators. We therefore expect a second payoff in back-of-head consistency.

Core Claim

Write the image position of canonical head point $X$ in frame $t$ as

$$x_t = \pi_f\big(s\,R_t\,(X + D(X, e_t)) + T_t\big),$$

where $\pi_f$ is pinhole projection with focal length $f$, $s$ is a per-video head scale, $R_t, T_t$ are head rotation and translation, $D$ is an expression-driven displacement, and $e_t$ is the frame's expression code. Scaling $s$ and $T_t$ together leaves every $x_t$ unchanged. Absolute size is therefore unobservable from monocular video. That leftover ambiguity is one scalar per video, fixed here by a learned prior on head size. The focal length $f$, by contrast, is observable from a rotating rigid shape at selfie distances, because the head's depth extent is a non-negligible fraction of its distance from the camera.

The claim has two parts. First, a model that predicts $f$ and $s$ once per video from all frames, while its identity and expression tokens are computed without any camera inputs (no Plücker ray encodings, no focal metadata), will keep canonical identity geometry invariant across focal and distance changes. Second, it will keep expression codes invariant to pure head translation. Camera-conditioned feed-forward avatars do neither. They also bear the cost of whatever focal length a monocular tracker assumes.

This is not the obvious composition of a metric depth model, a semantic anchor and a reconstruction transformer. We do not attempt metric depth. We argue it is the wrong target, because absolute size is unobservable. What we target is the factorization that tells the network where camera effects are allowed to live.

Research Program

We would build on the variable-length transformer design of FastAvatar, which reconstructs Gaussian-splat avatars (3D scenes made of many small colored blobs) feed-forward from one image up to a video, using FLAME mesh prompts and camera and expression guidance. It supplies the backbone. Our main change is its camera-conditioning pathway.

The video's frames pass through a shared image encoder. Identity tokens are pooled across frames with attention that has no access to camera or pose tokens. These tokens decode canonical Gaussians: FLAME-anchored Gaussians on the face, plus free Gaussians for hair and accessories, in the spirit of the hybrid that Gap 4 of the file calls for. Per-frame expression tokens drive a displacement field relative to the neutral canonical head. Predicting displacements from a neutral baseline is the approach Otberdout et al. use to avoid cross-identity stretching.

A separate small camera module sees 2D landmark tracks and image features from all frames. It outputs one $f$ and one $s$ per video, plus per-frame $R_t, T_t$. Only the differentiable renderer combines the two streams. The identity tokens cannot hear about the camera, so the only way to explain a lean toward the lens is through $T_t$. A wide-angle lens can only be explained through $f$.

Training needs data where the camera is known and varies. NeRSemble provides 222 subjects with 16 calibrated cameras, and Ava-256 provides 256 subjects with 80 cameras. Both let us recover metric head geometry by multi-view triangulation. From each subject we cut single-camera clips as monocular inputs. We also re-render reconstructed subjects at sampled distances and focal lengths, a perspective augmentation in the spirit of the canonical-camera normalization of Metric3D v2, which identified focal variation as the main driver of metric depth ambiguity. Supervision is photometric loss on held-out calibrated views, so we rely on multi-view rendering rather than 3DMM fitting, following the lesson of Portrait4D-V2.

Two further losses enforce the factorization. The first is a cross-clip identity consistency loss: two clips of the same subject captured at different simulated distances and focal lengths must produce the same canonical Gaussians. The second is a weak prior on $s$ fit to the triangulated head-size distribution. We will report honestly how much absolute-size error this prior leaves, since the theory says no image cue can remove it.

For the unseen back of the head, we follow the BalanceHead finding that view-invariant frontal conditioning removes directional bias. Because our identity tokens are already camera-free, we test whether this alone suppresses rear-face hallucination, before adding any generative prior.

Baselines

The naive reference is the same backbone with FLAME tracking from a standard monocular tracker at a fixed assumed focal length, and with its camera tokens fed this tracker output. This is the "do nothing about scale" pipeline that current monocular avatar methods implicitly use.

The strongest feed-forward competitor is FastAvatar itself. It is the closest system that decouples identity and expression feed-forward from variable-length input, and it conditions on camera pose and expression coefficients. We will retrain it, or use its released weights if they are available, on exactly our training data and augmentations, with the same compute and tuning budget. This ensures that any gain comes from the factorization and not from data. The file reports no FastAvatar numbers on INSTA or NeRSemble, so we will measure it ourselves. The file does report that its comparison method LAM falls to 16.49 PSNR as unaligned video frames are added, a warning that naive multi-frame aggregation degrades.

FastGHA is the few-shot calibrated reference, reporting 22.5 PSNR and 0.77 SSIM on Ava-256. We run it with tracker-estimated rather than ground-truth cameras to show what calibration is worth.

Among optimization-based monocular methods, the bar is set by FATE at 27.52 PSNR on INSTA and by ELITE at 25.220 PSNR, 0.8771 SSIM and 0.7396 identity cosine similarity (CSIM) in self-reenactment, with roughly 20 minutes of adaptation. INSTA covers the roughly 10-minute regime. We do not expect to beat FATE's photometric score without test-time optimization.

Beating the baselines means two things. First, matching FastAvatar's photometric quality within about 0.5 dB on held-out NeRSemble views. Second, clearly beating it on the factorization metrics defined below. Staying within roughly 2 dB of ELITE at a thousandfold lower latency would be a meaningful operating point.

The decisive ablation keeps our architecture but routes camera tokens into the identity and expression attention. A second ablation keeps the factorization but drops the cross-clip consistency loss. If either matches the full model, the principle is not doing the work.

Why not just…?

Why not just run Metric3D v2 per frame and unproject to metric points before reconstruction? Its canonical-camera transform needs the true focal length to return metric depth, and casual clips usually lack that metadata, as Metric3D v2 itself notes. Running it frame by frame also gives a fresh scale estimate each frame, which is precisely the time-varying signal that leaks into expression. We will still include it as a baseline that feeds per-frame depth to FastAvatar, and measure expression variance under pure translation.

Why not just augment FastAvatar's training with focal and distance variation and keep its architecture? This is the real threat. If camera conditioning plus augmentation already teaches invariance, our architectural change is unnecessary. The camera-tokens-in-identity ablation above, trained on identical augmented data, settles it. We will report the result whichever way it falls. If the gap is small, the contribution reduces to the augmentation and the evaluation protocol.

Why not just fix scale with an average inter-pupillary distance? Our formulation already admits that one scalar per video must come from a prior. The difference is where the prior acts. Applied to landmarks, it imposes a template size and moves with squinting or smiling. Applied to the per-video $s$ after identity is pooled, it rescales the whole head uniformly and cannot change face proportions or expression. We will compare canonical shape error after scale-only alignment for both choices.

Why not just optimize per person with INSTA or ELITE? These methods are more accurate today, and their monocular trackers face the same focal ambiguity. We report them as upper bounds on quality, and we test whether initializing them with our predicted $f$ and $s$ reduces their own shape error.

Proposed Experiments

The first decisive check is a dolly-zoom test on held-out NeRSemble and Ava-256 subjects. We re-render each subject at matched framing across a sweep of focal lengths and distances, then measure Chamfer distance between the predicted canonical heads after scale-only alignment. If identity is truly camera-free, this distance should stay flat across the sweep, while camera-conditioned baselines should track the sweep. Absolute head size error, in millimetres against triangulated geometry, is reported separately as the part only the prior can fix.

The second check is a lean test. In monocular clips where the subject translates toward the camera with a neutral face, we measure how much the predicted expression codes and the displacement magnitude vary. They should be near zero for our model.

The third check is focal recovery: error of the predicted $f$ against ground-truth calibration on held-out cameras. The file contains no competing focal-from-video estimator for heads, so we compare to a fixed-focal assumption and to FLAME-tracker focal optimization.

The fourth check is Janus: rendering held-out rear and profile views on NeRSemble with PSNR and LPIPS, plus a multi-face rate judged by a face detector on rear renders. We compare against BalanceHead's 360° behavior qualitatively, since its reported FID-random of 3.67 is on BalanceHead360 generation, not reconstruction.

Finally, we run cross-domain self-reenactment on INSTA monocular videos, reporting PSNR, SSIM, LPIPS and CSIM against FATE and ELITE, together with wall-clock reconstruction time.