Multi-view stereo networks recover 3D shape by testing many candidate depths for every pixel and asking which candidate makes the camera views agree. They store these comparisons in a "cost volume", a grid indexed by depth and pixel position. The grid grows with image resolution times the number of depth candidates, so it quickly exhausts GPU memory. Faces are a poor fit for this design in a second way. Facial tissue does not move along camera rays. It slides, stretches, and folds across the skull. The output is also an unstructured point cloud, while animation pipelines need a mesh with fixed connectivity. We propose to move the candidate search off camera rays and onto the face itself. Each vertex of a template mesh tests a handful of candidate positions. Those candidates lie along the directions that a learned non-linear face model says that vertex is likely to move. A recurrent network then updates the face model's identity and expression codes from the resulting matching costs.
The core claim has two parts. First, a cost volume indexed by mesh vertex and deformation hypothesis needs memory proportional to the number of vertices, not to image resolution. Second, sampling hypotheses along the non-linear model's local motion directions, rather than along surface normals or rays, is what lets a few iterations recover extreme and asymmetric expressions. The backbone question becomes a precise, testable sub-claim: a global-attention backbone matters mainly for the first guess that seeds the iterations, so it can run at low resolution while a cheap convolutional network supplies the high-resolution matching features.
Where this sits
Three lines of work each solve one piece of the problem. Efficient stereo methods tame memory by searching locally around a current estimate. Effi-MVS samples a few depth candidates around the current depth and refines them with a 2D convolutional recurrent unit. It reports 0.317 mm overall error on DTU at 3.1–3.6 GB of GPU memory. StereoNet computes matching on an 8×–16× downsampled volume and restores edges with image-guided refinement. We reuse their recurrent, local-window design. Both papers, however, parameterize candidates as fronto-parallel depth planes, and both admit that a bad coarse seed traps the local window. That limitation motivates our seeding study.
Non-linear face models supply the deformation space. CoMA uses graph convolutions on a fixed-topology mesh and roughly halves error on extreme expressions compared with linear blendshapes. The 2500 FPS mesh decoders show that such decoders are cheap enough to call inside an inner loop. ImFace and ImFace++ model faces as signed distance functions, each driven by an implicit network. They disentangle expression and identity deformations and reach 0.511 mm Chamfer distance and 96.11% F-score at 1 mm on FaceScape. Fitting them takes hours per sample, and they produce no fixed topology. We borrow their identity/expression split but use an explicit mesh decoder so that the vertices can index a cost volume.
The closest hybrid is topology-uniform face meshes from volume rendering. It optimizes the vertices of a MetaHuman template, together with feature planes that store color, against 30 calibrated views. It reaches 3.255 mm mean error, against 3.725 mm for Metashape followed by non-rigid ICP. It works per subject, with no learned deformation space and no cross-view matching volume. FNeVR is fast but handles a single view and relies on a linear FLAME prior. On the backbone side, the evidence that CNN features cap downstream modules comes from other tasks: Locate-Then-Segment, UniInst, and Orthogonal Single-Target Tracking. We treat it as a hypothesis to test in geometry, not as settled.
Mechanism
A non-linear mesh decoder $G(z_{id}, z_{exp})$ produces the positions $x_v$ of $V$ template vertices. It is trained on FaceScape's topologically uniform registered meshes, with separate identity and expression codes and a per-vertex detail residual.
At iteration $t$ the network forms $K$ candidate positions for each vertex:
$$h_{v,k} = x_v + J_v U_v \epsilon_k + \sigma_t n_v \eta_k.$$
Here $J_v$ is the decoder's Jacobian at vertex $v$, which says how that vertex moves when the latent codes change. $U_v$ holds that Jacobian's top few right-singular directions, the latent directions that move vertex $v$ the most. The terms $\epsilon_k$ and $\eta_k$ are fixed sampling offsets, $n_v$ is the surface normal, and $\sigma_t$ is a shrinking normal-direction radius. The Jacobian term gives tangential, expression-shaped candidates. The normal term covers detail the decoder cannot express.
Each candidate is projected into all views. Its features are compared using group-wise correlation variance, giving a cost tensor of size $V \times K \times C$, where $C$ is the number of correlation groups. A recurrent unit built from mesh convolutions reads this tensor and outputs updates to $z_{id}$, $z_{exp}$, head pose, and the detail residual. The cost of one iteration is
$$M_{ours} \propto V K C \quad \text{versus} \quad M_{MVS} \propto D H W C.$$
The planar cost $M_{MVS}$ grows with depth planes $D$ times image size $H \times W$. With $V \approx 10^4$ and $K \approx 32$, our volume is independent of image resolution, so the matching features can be sampled from full-resolution feature maps.
The seed codes come from a separate encoder. That encoder is where the backbone study lives.
Research Program
The work proceeds in three steps.
The first step trains the deformation decoder and verifies that it is expressive on its own. We fit it directly to held-out FaceScape scans and compare with FLAME and with ImFace's reported numbers. This tells us whether the decoder itself caps final accuracy.
The second step builds the deformation-space cost volume and the recurrent updater. They are trained on multi-view images rendered from FaceScape textured scans under randomized cameras and lighting. Supervision is 3D vertex loss plus photometric consistency.
The third step runs the backbone study. We separate two roles a backbone plays: producing the seed and producing the matching features. Each role is filled independently with a ResNet-50, a ViT, or a hybrid. For every combination we measure seed error, final error, how often the iterations fail to escape a bad seed, and peak memory. The hypothesis predicts that a low-resolution ViT seed combined with high-resolution CNN matching features reaches the accuracy of using the ViT everywhere, at a fraction of the memory. If instead the matching features need global attention too, the hypothesis is refuted.
Baselines
The naive reference point is the seed alone: the encoder's predicted codes decoded once, with no cost volume and no iterations. It is run with the same backbone, training data, and decoder. Anything the iterations add has to be measured against it.
The strongest geometry competitor without fixed topology is Effi-MVS. It is trained on the same rendered FaceScape multi-view data with the same budget, and its fused point cloud is registered to the template by non-rigid ICP. This matches the pipeline Wang et al. report degrading to 3.725–4.412 mm. Its published DTU result, 0.317 mm overall error at 3.1–3.6 GB, does not transfer to faces, and no reported result exists on Lacuna for Effi-MVS on face data. We therefore report it under our protocol. Beating it means lower Chamfer error on the final mesh at equal or lower peak memory.
The strongest topology-preserving competitor is the per-subject optimization of Wang et al.. On their 30-view dataset it reports 3.255 mm mean geometric error and 32.10 novel-view PSNR. If that dataset is accessible we evaluate there. Otherwise we run their method on our rendered views. Beating it means matching its error within about 0.2 mm while running feed-forward in seconds instead of per-subject optimization.
For the upper bound on deformation-model expressiveness we cite ImFace++ on FaceScape: 0.511 mm Chamfer distance and 96.11% F-score at 1 mm, from fitting to 3D scans rather than images. It is a ceiling for the representation, not a like-for-like competitor.
Two ablations isolate the key ingredient. The first, the normal-only variant, drops the Jacobian term from the candidate equation and searches only along surface normals. The second, the linear variant, swaps the non-linear decoder for FLAME's linear bases, so the Jacobian is constant. Both share every other component and the same tuning budget. The claim holds only if the full model beats both, with the gap concentrated on extreme and asymmetric expressions.
Why not just…?
Why not just regress the non-linear codes directly from a strong ViT, with no cost volume? This is the most serious threat. If global features already pin down the face, matching adds nothing. The seed-only baseline answers it directly. We expect regression to get identity roughly right but to miss metric-scale detail and asymmetric expressions, because no cross-view check enforces them. That expectation is consistent with Coarse-to-Fine 3D Face Reconstruction, which finds that learned 2D refiners inherit errors from their input estimate. If the seed-only model comes within 0.1 mm of the full model on FaceScape, we would drop the iterative stage and report the backbone finding alone.
Why not just search along surface normals, which is ordinary mesh-anchored stereo? This is also a real threat. Many facial errors after a decent seed may be normal displacements, which is exactly what the normal-only ablation tests. Our argument is that expression motion is largely tangential, the observation behind CoMA. A purely normal search will therefore stall on mouth corners, cheeks, and eyelids during extreme expressions. We report per-region and per-expression error so the outcome is visible either way. If the normal-only variant ties, the honest conclusion is that the deformation model matters as a seed and a regularizer, not as a search space.
Why not just run a linear 3DMM inside the same pipeline? The evidence that linear spaces are limiting is strong. FLAME reaches 0.971 mm Chamfer distance on FaceScape, against 0.625 mm for ImFace. The part-guided method of Wang et al. reaches 1.436 mm on REALY yet still reports being capped by its linear basis. A linear basis also makes $J_v$ constant, so its candidate directions cannot depend on the current expression. The linear ablation measures how much this costs once the free per-vertex detail residual can absorb part of the gap.
Proposed Experiments
We train on FaceScape subjects with images rendered from many viewpoints and test on held-out subjects and expressions. Cross-dataset tests use renders from BU-3DFE and LYHM scans, and the 30-view capture dataset when available. We report Chamfer distance, F-score at 1 mm, and normal consistency against scans, plus per-region error. For the 30-view data we also report mean geometric error and novel-view PSNR. Peak GPU memory and runtime are reported at 1K, 2K, and 4K input resolution.
Three checks decide the outcome. First, memory must stay flat as input resolution grows for our method, while the Effi-MVS baseline's memory grows. Second, the full model must beat the normal-only and linear ablations specifically on the hardest expressions in FaceScape's expression set. Third, in the backbone study, the hybrid with a low-resolution ViT seed and high-resolution CNN matching features must match the all-ViT model within noise. We also expect seed-failure rate, not matching quality, to explain most of the gap between CNN and ViT seeds. If any of these fails, the corresponding part of the claim is withdrawn, and the remaining parts still form a clean report on where facial stereo error comes from.
References
- Efficient Multi-view Stereo by Iterative Dynamic Cost Volume
- StereoNet: Guided Hierarchical Refinement for Real-Time Edge-Aware Depth Prediction
- Generating 3D Faces Using Convolutional Mesh Autoencoders
- Dense 3D Face Decoding Over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders
- ImFace: A Nonlinear 3D Morphable Face Model with Implicit Neural Representations
- ImFace++: A Sophisticated Nonlinear 3D Morphable Face Model With Implicit Neural Representations
- Learning Topology Uniformed Face Mesh by Volume Rendering for Multi-view Reconstruction
- FNeVR: Neural Volume Rendering for Face Animation
- Locate Then Segment: A Strong Pipeline for Referring Image Segmentation
- UniInst: Unique representation for end-to-end instance segmentation
- Orthogonal Single-Target Tracking
- Coarse-to-Fine 3D Face Reconstruction
- 3D Face Reconstruction with the Geometric Guidance of Facial Part Segmentation