Frontier Points as Drift-Free Anchors for Monocular Face Tracking

When a head turns in front of a single camera, the outline of the cheek and jaw does not stay on the same patch of skin. The outline sits wherever the surface curves away from the camera, so as the head rotates the outline slides across the face. Current face trackers treat this sliding as a nuisance to correct one frame at a time. They then face two further problems on their own: a single image cannot pin down how deep the face is along the camera's line of sight, and small per-frame errors compound over long videos. This proposal argues that the sliding outline can help with both. For any two frames where the head has turned enough, there exist a few outline points that really are the same physical point in both views. Classical multi-view geometry calls these frontier points. They can be located from the two outlines and the relative head pose alone, with no appearance matching, and once located they can be triangulated in 3D. A tracker that collects these points between widely separated frames gets geometric constraints on face depth that no single frame provides. Because the constraints come from geometry rather than from remembered appearance, they keep working when the head shows the camera surfaces that were hidden in the first frame.

Put in two sentences: the outline of a rotating face is not a fixed landmark, but between any two sufficiently different views it touches the same 3D point at a computable location. We use those points as long-range anchors that fix depth and stop drift in monocular video face reconstruction.

Core Claim

The claim is that frontier-point constraints between non-adjacent frames add information that existing approaches lack. Per-frame dynamic-silhouette fitting does not have it, temporal smoothing does not have it, and appearance-based anchoring to a reference frame does not have it. Adding these constraints to a video face tracker should reduce two errors that the field currently conflates. The first is depth bias: the reconstructed face is too flat or too wide along the viewing direction. The second is drift: error grows with frame index during long sequences that include large yaw rotations. We also predict where the gain should appear. It should be concentrated in the cheek, jaw and temple regions, and in sequences whose yaw range exceeds roughly 30°. Below that range the two-view baseline is too short for triangulation to be informative.

Why This Gap Is Real

The dynamic-silhouette literature solves correspondence one frame at a time. Asthana et al. look up pose-dependent vertex indices, and Zamuner et al. update the shape model during tracking. Dense Face Alignment fits the model boundary to an edge-detected outline and reports a 28% improvement at 60°–90° yaw on AFLW2000-3D. Shan et al. re-assign silhouette landmarks during coarse fitting to prevent profile widening. Mao et al. select boundary vertices by the condition that the surface normal is perpendicular to the viewing ray, running at 30 FPS. All of these papers motivate the problem, and the last one supplies our per-frame machinery. None of them uses the outline in one frame to constrain geometry in another.

These methods share a weakness that the research file names directly. If depth or out-of-plane rotation is wrong, the perpendicularity test selects the wrong vertices. The resulting contour loss then pulls the pose further toward the wrong answer. In a tracker that carries state forward, this loop compounds along the camera axis.

Video methods attack depth ambiguity differently, and they constrain our design. FML enforces a constant identity shape and skin color across frames of a clip and reaches 1.74 mm error on BU-3DFE. Its photometric loss, however, assumes that a pixel near the cheek outline maps to the same skin point in every frame. That assumption is exactly wrong at the boundary.

The tracking literature supplies the vocabulary of drift. STM showed that a mistake committed to a memory of past frames poisons later frames. FEELVOS and Anchor Diffusion anchor every frame to the first one by appearance. That fails when rotation exposes surfaces never seen in the anchor frame. Gating approaches, namely the Selector Net and detector-in-the-loop rectification, reject bad updates using an independent check. We borrow that idea and use a geometric residual as the check.

The method itself comes from older multi-view work. Generalised Epipolar Constraints showed that frontier points behave as true correspondences between two views of a smooth object and used them to recover rigid camera motion. Vogiatzis et al. used frontier points as anchors whose 3D position and normal are known from outlines alone, reaching 4.6%–10.1% mean depth error on figurines. Both assumed rigid objects, calibrated or pairwise-chained cameras and multi-view capture. Neither was applied to a deformable face model tracked in monocular video, and neither used frontier points across non-adjacent frames to counter drift. That transfer is the contribution here.

Research Program

The starting point is a per-frame regressor that predicts the parameters of FLAME, a parametric head mesh with shape, expression and pose parameters. It is trained with the normal-based silhouette vertex loss of Mao et al.. A face-parsing network supplies an observed 2D outline in every frame. On top of this regressor we build a sliding-window optimizer that refines pose, expression and a shared identity shape over a set of keyframes.

Keyframes are selected by geometric baseline rather than by appearance similarity. A frame pair qualifies when the estimated relative head rotation exceeds a threshold. We treat the head as fixed and the camera as moving around it. A head rotation about the neck then corresponds to a virtual camera translation, because the camera is offset from the rotation axis, and that translation is what makes triangulation possible.

For each qualifying pair $(i, j)$, we use the current relative pose to compute the epipolar lines, which are the lines in each image along which a point seen in the other image must lie. We then find the points $x_i$ and $x_j$ where these lines are tangent to the observed outlines. Triangulating those two points gives a 3D point $X_{ij}$. The mesh is penalized with

$$ L_{\text{fr}} = \sum_{(i,j)} w_{ij}\left[\, d\big(X_{ij}, S\big)^2 + \lambda\,\big(\mathbf{n}_S(X_{ij})\cdot \mathbf{b}_{ij}\big)^2 \right], $$

where the symbols mean the following. $S$ is the identity-plus-expression mesh in head coordinates. $d$ is point-to-surface distance. $\mathbf{n}_S(X_{ij})$ is the mesh normal at the closest surface point. $\mathbf{b}_{ij}$ is the unit baseline between the two virtual camera centers. $\lambda$ balances the two terms. $w_{ij}$ is a confidence weight that shrinks when the baseline is short or the epipolar line meets the outline at a shallow angle, both of which make triangulation unstable.

The first term says that the frontier point must lie on the surface. The second term says that the surface there must be tangent to the plane containing both camera centers. Pose enters through the epipolar geometry, so the loss constrains relative rotation and depth jointly. This is exactly the coupling that per-frame contour fitting gets wrong.

Faces are not rigid, which requires two restrictions. Frontier constraints are applied only in regions whose motion is dominated by the shared identity shape: cheek, temple and upper jaw. Pairs whose expression codes differ beyond a threshold are discarded. An ablation measures how much accuracy the second restriction costs.

Two further pieces complete the system. First, we fix FML-style multi-view photometric consistency by masking a band around the predicted contour generator in each frame, so that boundary pixels never vote for false correspondences. Second, the per-pair frontier residual serves as a drift detector. When the residual for a pair involving the newest frame exceeds a threshold, that frame is not admitted as a keyframe and the window is re-solved from older keyframes. This adapts the gating idea of the Selector Net to geometry instead of learned mask quality.

Baselines

The naive reference point is the per-frame regressor run independently on each frame, with a static, fixed-index assignment of boundary landmarks to mesh vertices. It shows how bad the problem is without any dynamic silhouette handling. Dense Face Alignment documents that this choice fails at large yaw.

The strongest per-frame competitor is the base model of Mao et al., including its person-specific adapter fine-tuning on the test subject's other frames. It combines dynamic silhouette handling with multi-frame identity adaptation, which makes it the closest existing system to ours. We run it with the same backbone and FLAME version, the same face parser and the same frames for adaptation, and we give it a tuning budget equal to ours. The file reports no numbers for it on 300-VW or MICC ("no reported result found on Lacuna for Mao et al. on 300-VW or MICC"), so we reproduce it.

The strongest multi-frame competitor is FML, which reports 1.74 mm reconstruction error on BU-3DFE. We evaluate it, and our system, on BU-3DFE renderings to check that our numbers are in the same regime. On synthetic rotation sequences we run FML with the same windows our optimizer sees.

For single-image geometric accuracy, we take as reference points Shan et al. at 1.56 mm 3DRMSE on Bosphorus and the FaceScape single-view result of 1.22 mm mean point-to-surface distance reported by Zhu et al..

The key ablation keeps everything except $L_{\text{fr}}$: the same windowed optimizer, the same keyframes, the same masked photometric loss and the same shared identity. Any gain from the full system over this ablation is attributable to frontier points alone. A second ablation selects keyframes by appearance similarity rather than by baseline.

Beating the baselines means two things on the synthetic rotation set, each at least as large as the variation across five random seeds. The cheek-and-jaw region error must fall below the Mao et al. baseline. The slope of error against frame index must be lower than both the smoothed baseline and the frontier-free ablation. Matching them on global error while losing on either regional error or slope would count as a failure of the core claim.

Why not just…?

Why not just run the per-frame silhouette regressor and smooth its outputs over time? This is the most serious threat. A Kalman or one-euro filter on the FLAME parameters is cheap and will remove jitter, and on 300-VW-style stability metrics it may match us. Smoothing, however, averages per-frame estimates that share the same bias. If every frame flattens the face along the viewing ray, the smoothed face is flat too. Smoothing also cannot correct the pose-to-contour feedback loop, because the loop operates inside each frame's fit. We include the smoothed regressor as a baseline and separate jitter from bias by reporting temporal-acceleration error alongside per-region geometric error against ground truth. If smoothing closes the regional depth gap on the synthetic set, the core claim is wrong and we will say so.

Why not just mask the boundary in FML's photometric loss and stop there? Masking removes the false correspondences, but it also discards the outline information entirely. What remains is interior shading and landmarks, which are exactly the cues that leave cheek depth weakly determined. This masked variant is the frontier-free ablation in our program, so the experiment directly measures whether putting the outline back as frontier points adds anything beyond removing its harm.

Why not just anchor every frame to the first frame, as FEELVOS or Anchor Diffusion do? Appearance anchoring needs the anchor to show the same surface. A head turning from frontal to profile exposes ear and cheek regions that are absent from frame zero. The FEELVOS ablation also shows that global matching alone drops J&F on DAVIS 2017 from 69.1% to 54.9%, so purely global anchors are weak even when appearance is shared. Our anchors are chosen because the views differ, which is the reverse of appearance anchoring. The appearance-keyframe ablation tests this choice directly.

Why not just add a differentiable-rendering mask overlap loss across frames? Aligning Silhouette Topology shows that pixel-overlap silhouette losses give nearly flat gradients and invite shape inflation in place of correct pose. A multi-frame overlap loss inherits both problems. We include it as an additional baseline at equal compute.

Proposed Experiments

The controlled testbed is a set of synthetic monocular videos rendered from FaceScape scans (Zhu et al.) and BU-3DFE scans. Each video follows a scripted yaw sweep with amplitude between 10° and 90°, and some sequences include expression changes and realistic camera intrinsics. Ground-truth geometry is known in every frame. On this set we measure three things: per-region point-to-surface error, split into interior versus cheek/jaw/temple; error of the estimated face depth along the optical axis; and drift, defined as the fitted slope of error against frame index over sequences of 300 or more frames.

The first decisive check is the yaw-amplitude curve. Our method should match the frontier-free ablation below about 30° yaw and pull ahead above it, because that is where triangulation becomes well-conditioned. If the gain does not grow with yaw amplitude, the mechanism is not what we claim.

The second decisive check is a pose-perturbation study. We inject known out-of-plane rotation errors into initialization and test whether $L_{\text{fr}}$ pulls pose back while the per-frame contour loss amplifies the error, the feedback loop the research file hypothesizes.

On real data, we use MICC videos for 3D accuracy against its scans and 300-VW for 2D landmark stability. On 300-VW we expect parity rather than a large gain, because 2D metrics hide depth error (Cascaded Error Propagation). Bosphorus and AFLW2000-3D serve as single-image sanity checks, confirming that the per-frame base model has not been degraded.

Finally, we evaluate the drift detector on its own by checking whether the frontier residual predicts frames whose ground-truth error is in the top decile. A useful detector would justify reusing it as a memory gate for appearance-based trackers such as STM.