A single video of a face does not say how far each point of the face is from the camera. A smile that pushes the cheeks forward and one that spreads them sideways can produce nearly the same image. Current pipelines that drive a 3D character from ordinary video settle this ambiguity frame by frame, usually by snapping the face to a learned shape prior. They then copy the result onto the character's mesh through vertex correspondences. Both steps are brittle. The prior invents depth it cannot see, and vertex-indexed transfer breaks when the character has a different mesh layout, or has separate pieces for eyes, teeth and tongue.
We propose to treat retargeting as a control-theory problem. The quantity being tracked is a continuous 3D deformation field defined on the space around the face, not on any one mesh's vertices. That field is estimated from 2D image motion by a nonlinear observer, meaning an estimator that combines a model of how the state evolves with partial measurements, in the spirit of a Kalman filter for nonlinear systems. The central idea is that depth which is unobservable in one frame often becomes observable over a short window, because the head rotates and expressions follow constrained nonlinear dynamics. Nonlinear observability analysis can say in advance which depth components a given clip can recover and which it cannot. The system then commits only to recoverable depth and leaves the remainder to the target character's own prior. Because the estimated object is an ambient-space field, the same estimate can deform any target mesh regardless of its vertex count or connectivity.
Core Claim
Monocular depth ambiguity in facial capture is structured. It is concentrated in a few deformation directions, and how many of those directions are hidden depends on the head motion and expression dynamics in the clip. An observer built on a topology-free deformation state can recover depth in the directions that the clip makes observable. In those directions it should beat per-frame reconstruction followed by temporal smoothing. In the directions that remain unobservable, it should state its own uncertainty correctly instead of hallucinating.
The new element is not the topology-free field, and not monocular face capture alone. It is the use of an observability calculation to decide, per clip and per deformation direction, what monocular retargeting is entitled to claim.
Relation to Prior Work
Two papers in the file supply the topology-agnostic machinery, and one of them is also the closest competitor.
Defoxel models shape editing as a force field in ambient space grounded in solid mechanics, which separates the deformation from any particular polygon layout. We reuse that principle, deforming space rather than vertices, as the state representation. We differ in that our field is estimated from video rather than authored by an artist.
What Makes Tom Hanks Look Like Tom Hanks (also indexed as art_595c1862; What Makes Tom Hanks Look Like Tom Hanks) builds controllable 3D persona models from unstructured photo collections. It estimates 3D flow from driving frames and maps those displacements onto target geometry, separating identity from expression. This is the closest existing work and the direct motivation for our proposal: it shows that flow-based transfer from 2D imagery to a target shape is workable. As we read it, it resolves depth inside its reconstruction and does not report which components of the transferred flow were actually supported by the images. Our proposal makes that distinction explicit and acts on it.
The research file also mentions intrinsic surface networks and neural Jacobian fields as topology-agnostic alternatives. It gives no citable entries for them, so we treat them only as possible replacements for the field parameterization.
The file contains no section on prior combinations of these directions and no benchmark entries. We found no work on Lacuna that combines nonlinear observability with facial retargeting, but that search was limited to what the file contains.
Research Program
State. The state at time $t$ is $x_t = (c_t, R_t)$, where $c_t$ holds the coefficients of a smooth ambient-space displacement field around a canonical head and $R_t$ is the rigid head pose. We will use a compact basis for the field, such as radial basis functions on a coarse lattice, so that it can be evaluated at any point in space.
Dynamics. Expression dynamics are modeled as $c_{t+1} = f_\theta(c_t) + w_t$. Here $f_\theta$ is a small learned nonlinear transition model fitted on multi-view capture, and $w_t$ is process noise. Its role is to encode that real faces do not move arbitrarily: muscles couple regions, and motion has inertia.
Measurements. Measurements are 2D image flow and landmarks, $y_t = \pi(R_t \, \Phi(c_t)) + v_t$. In this expression, $\Phi$ maps field coefficients to displaced canonical surface points, $\pi$ is the camera projection, and $v_t$ is measurement noise.
Observability. Over a window of $T$ frames we compute the empirical observability Gramian:
$$W_T = \sum_{t=0}^{T} J_t^\top J_t, \qquad J_t = \frac{\partial y_t}{\partial x_0}.$$
Here $J_t$ is the sensitivity of the frame-$t$ measurement to the initial state, obtained by differentiating through $f_\theta$ and $\pi$. Field directions where $W_T$ has small eigenvalues are directions the clip cannot pin down.
Estimation and fallback. The observer is an iterated extended Kalman smoother. It returns a posterior over $c_t$. Along poorly observed directions the estimate falls back to a prior learned on the target character's own expression range, not the source actor's. The estimated field is then evaluated at the target mesh's vertices, at every connected component, so eyes, teeth and lashes move consistently without any correspondence. A light correction step keeps disconnected parts from interpenetrating.
Analysis. The analytical part of the work characterizes which head motions make which expression modes observable. Our hypothesis is that rotations about the vertical axis expose forward cheek and lip motion, while a frontal static head leaves jaw protrusion largely hidden. This gives practitioners a capture protocol, not just an estimator.
Baselines
The naive reference is per-frame monocular reconstruction with a fixed-depth assumption: 2D flow is lifted using the canonical head's depth and transferred through the field, with no temporal model. Any method that cannot beat it on depth error is not using the video.
The strongest existing competitor is per-frame 3D reconstruction with a learned shape prior, followed by flow transfer onto target geometry, in the manner of What Makes Tom Hanks Look Like Tom Hanks. We will reimplement this pipeline inside our own field representation, so the only difference from our method is how depth is decided. Both systems will use the same training captures, the same image features, and the same per-method hyperparameter search budget. We found no reported result on Lacuna for this method's depth or retargeting error, so its numbers will come from our own runs. Beating it means lower depth error along well-observed directions on held-out clips. It also means calibrated uncertainty along poorly observed directions: the observer should do no worse there than the prior, while flagging those directions correctly.
The key ablation keeps the observer and the field but removes the learned dynamics, replacing $f_\theta$ with a random walk. If this ablation matches the full system, the gain comes from temporal pooling of head rotation alone, and the nonlinear dynamics claim fails.
A second ablation replaces the field with vertex-indexed transfer on targets whose topology matches the source. This isolates the cost or benefit of the topology-free representation where correspondence is available.
Why not just…?
Why not just smooth per-frame reconstructions over time? Smoothing averages estimates that may all share the same biased depth from the prior. It cannot recover a direction that no single frame constrains. The observer instead uses the geometry of head rotation across frames, which is information smoothing never sees. The test is direct. On clips with substantial head rotation, the observer should beat the smoothed baseline in depth error along directions that per-frame Gramians rate as unobservable but window Gramians rate as observable. If it does not, this alternative wins and our proposal reduces to a capture protocol.
Why not just trust a strong learned 3D face prior? This is a real threat. On in-distribution expressions from frontal video, a good prior may match the observer. The prior's failure mode is out-of-distribution motion: exaggerated performances, asymmetric expressions, and actors unlike the training population. In those cases it produces confident but wrong depth. We will report results split by in-prior and out-of-prior expressions. Our claim is restricted to the latter, plus calibration everywhere.
Why not just retarget with sparse landmarks and thin-plate warping? Landmark warping is topology-agnostic and cheap. However, it transfers only what the landmarks see and inherits whatever depth the landmark detector assumes. It neither resolves nor reports ambiguity. It will be included as an additional reference on stylized targets, where its simplicity may make it competitive in perceptual quality even if it loses on geometric error.
Proposed Experiments
The file lists no benchmarks, so evaluation will use a multi-view facial capture set. One camera is held out as the monocular input, and the remaining cameras provide reconstructed ground-truth depth and deformation. Clips will be grouped by measured head-rotation range and by whether their expressions fall inside the prior's training distribution.
Metrics are:
- depth error along the camera axis, decomposed into Gramian-ranked well-observed and poorly observed directions;
- surface error of the retargeted result, measured against ground-truth deformation of a same-topology proxy;
- frame-to-frame jitter;
- expected calibration error of the posterior.
Topology-agnostic transfer will be evaluated on target meshes with differing vertex counts and with disconnected eye, teeth and lash components. We will report interpenetration rates, and for stylized characters without ground truth we will run a blinded perceptual preference study.
The decisive check is whether the observability Gramian, computed before seeing ground truth, predicts where every method fails. Per-direction error should fall as the Gramian eigenvalue rises, for the observer and for the baselines alike. Among well-observed directions, the observer's advantage over smoothed per-frame reconstruction should grow with head-rotation range. If error is unrelated to predicted observability, the core claim is wrong, whatever the aggregate scores show.