Joint Rigid Alignment and Morphable Reconstruction from Partial 3D Face Scans

Eli A. Underwood, Nicolas Rondan, Angel V. Peterchev, J. Matias Di MartinoarXiv 2026 · arXiv

TL;DR

Researchers have developed a single-network architecture that jointly estimates the rigid pose and completes the missing geometry of partial 3D face scans in a single forward pass. By coupling a 3D morphable model (3DMM) directly with a geometric transformer, the method avoids the "shape collapse" common in generic registration tools and achieves 3–4× better accuracy while being significantly faster than current state-of-the-art baselines.

What Problem Does This Paper Address?

Capturing 3D faces with handheld or single-shot scanners usually results in "partial" data—you might get a good view of the nose and cheeks, but the ears, cranium, and far side of the face are missing. To make this data useful for things like prosthetic design or animation, you have to solve two "chicken-and-egg" problems simultaneously:

  1. Rigid Alignment: Finding the rotation and translation to move the scan into a standard reference frame.
  2. Non-Rigid Completion: Filling in the missing parts of the head based on prior knowledge of human anatomy.

Existing pipelines usually decouple these. They run a generic point-cloud registration tool first, then try to fit a face model. The authors point out a critical flaw in this approach: generic non-rigid solvers often satisfy their "smoothness" requirements by flattening or collapsing the head onto the visible scan area rather than reconstructing a plausible 3D shape. This creates a "blind spot" where standard metrics like Chamfer Distance look good (because the visible points match), but the actual reconstructed anatomy is a disaster.

Figure 1. Mean-face template and partial validation scan (top), with the morphed references produced by three representative non- rigid registration baselines—OAReg [26], ClusterReg [25], and CPD [11]—and our method (bottom). The baselines collapse the head onto the visible region, whereas our method recovers a plau- sible full head from the same partial input.

Methods

The core innovation is an end-to-end architecture that shares geometric features across the matching and reconstruction stages.

  1. Feature Extraction & Coarse Matching: The system uses a KPConv backbone and a GeoTransformer to identify "superpoint" correspondences between the partial scan and a generic mean-face template.
  2. Cross-Attention Morphable Regressor: Instead of fitting a global model, the network predicts coefficients for 32 independent local patches of a 3D Morphable Model (learned from 8,000 identities). This regressor is conditioned on the transformer's features and the predicted correspondences. This allows the network to "know" which parts of the face are observed and which need to be hallucinated.
  3. Morphed Reference Matching: The predicted coefficients deform the template into a scan-specific "morphed reference." This morphed version is then used as the target for fine-grained Sinkhorn matching to finalize the rigid pose.
  4. Scan-Referenced Refinement: A final, short optimization step (warm-started by the network's prediction) snaps the visible regions of the morphed template tightly to the actual scan data without losing the anatomically correct shape in occluded areas.
Figure 2: The end-to-end pipeline overview showing the joint matching and morphing process.

Datasets

The model was trained and evaluated using:

  • Synthetic Corpus: 8,000 synthetic head identities sampled from the Universal Head Model (UHM). 7,500 were used for training, and 250 were held out for a rigorous validation set featuring "Face-Cut" and "Self-Occlusion" crops.
  • NoW (Not-quite-in-the-Wild): A benchmark of real human scans used to test generalization to real-world data.
  • NPHM (Neural Parametric Head Models): An independent real-scan dataset used for further validation.

Key Insights and Ideas

  • The Chamfer Blind Spot: The authors demonstrate that Chamfer Distance (CD) is a poor proxy for reconstruction quality on partial scans. Methods can achieve low CD by "cheating"—collapsing the entire head mesh into a thin sheet that sits on top of the scan.
  • Coupling is Key: By regressing morphable coefficients during the matching process, the rigid alignment becomes "shape-aware." The pose is estimated against a target that already looks like the specific individual, not just a generic average human.
  • Local Basis vs. Global Basis: Using 32 local patches instead of one global PCA basis allows the network to exploit local evidence. If the scan only sees a nose, the local patch for the nose is constrained by data, while the back of the head remains governed by the learned anatomical prior.

Key Results

  • Accuracy: On the validation set, the method achieved a median per-vertex error of ~2.4mm, compared to 8.5mm–46.6mm for various baselines.
  • Speed: The method is remarkably efficient, processing scans in approximately 2.5 seconds. This is roughly 6× faster than the strongest baseline (NSFP) and up to 25× faster than classical methods like NRICP.
  • Robustness: As shown in Figure 3, the method "Pareto-dominates" every baseline, offering a better trade-off between accuracy and latency across all tested compute budgets.
Figure 3. Accuracy (1-way CD, partial scan→deformed-template, median) vs. wall-clock time (log scale, includes rigid-ICP pre- alignment for every baseline that requires it) across each method’s own iteration/step budget, NoW benchmark test split. Baselines: NRICP [1], CPD [11], NSFP [10], GraphSCNet [16], Cluster- Reg [25], OAReg [26].

Limitations and Future Work

  • Basis Expressiveness: The morphable prior is a linear PCA basis. While this prevents shape collapse, it also caps the model's expressiveness. The authors note that under-represented identities (specific ages or expressions) might not be perfectly reconstructed.
  • Synthetic-to-Real Gap: While the model performs well on real scans (NoW/NPHM), it was trained entirely on synthetic data.
  • Hair and Occlusions: The current template is hairless. Real-world scans with significant hair or non-facial occlusions (like glasses) remain a challenge for evaluation, as ground truth for the underlying skull shape is rarely available in those datasets.

Summary updated

Abstract

Aligning a partial, real-world 3D face scan to a canonical representation requires solving two coupled problems at once: estimating the rigid pose that brings the scan into a common frame, and completing the facial shape in regions the scan never observed. Existing pipelines typically decouple these steps, running a generic point-cloud registration front end followed by a separate non-rigid fitting or deformation stage. Building on the GeoTransformer superpoint-matching architecture, we propose a single network that predicts both a rigid pose and a shape-complete point cloud in one forward pass: a cross-attention regressor, conditioned on transformer point features and predicted coarse correspondences, maps a partial scan to per-patch coefficients of a 3D morphable face model learned offline from a corpus of 8,000 identities. The resulting morphed template is then used, in the same forward pass, as the target for fine-grained Sinkhorn matching and rigid pose estimation, supervised purely by geometric alignment to ground-truth morphed geometry, so no hand-labeled coefficients are required. Against six classical and learned non-rigid registration baselines on held-out identities, our method leads the strongest ones by 3-4 times on per-vertex error and baselines that collapse the head onto the visible region by more than an order of magnitude, and is simultaneously the most accurate and fastest method on ground-truth-free scans from two independent real-scan datasets, NoW and NPHM. Code will be made publicly available.
BibTeX
@misc{underwood2026joint,
  author = {Eli A. Underwood and Nicolas Rondan and Angel V. Peterchev and J. Matias Di Martino},
  title = {{Joint Rigid Alignment and Morphable Reconstruction from Partial 3D Face Scans}},
  year = {2026},
  eprint = {2610.39161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39161},
}
Figure