Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration

Xizhi Wang, Yaxiong Wang, Mengjian Li

TL;DR

The paper introduces Geometric-Photometric Joint Alignment (GPJA), a novel method for tracking and registering 3D facial expressions with pixel-level accuracy. By leveraging differentiable rendering to combine 3D geometric data with dense 2D photometric appearance, GPJA automatically produces topologically consistent meshes with a shared texture parametrization across diverse expressions without requiring manual annotations or extensive training data.

What Problem Does This Paper Address?

Creating high-fidelity, topologically consistent 3D head meshes across different facial expressions is a foundational requirement for modern movies, video games, and VR applications. Traditional registration pipelines typically rely on non-rigid Iterative Closest Point (ICP) algorithms to warp a generic template mesh onto a 3D scan using sparse landmarks or 2D optical flow.

However, these approaches often ignore dense, pixel-level photometric (color) consistency. This oversight leads to texture misalignment—such as a mole, wrinkle, or lip contour shifting unnaturally across different expressions—which requires tedious, manual frame-by-frame clean-up. While recent deep learning methods attempt to automate this, they often struggle to capture fine-grained details and require massive, pre-aligned datasets for training.

Common errors encountered when relying solely on landmarks or optical flow under large deformations.

GPJA aims to solve these challenges by jointly optimizing both geometry and appearance automatically, ensuring that the same physical point on a subject's face maps to the exact same texture coordinate regardless of the expression.

Methods

GPJA avoids explicit, error-prone correspondence matching by treating registration as an inverse rendering problem. Guided by multi-view images and 3D scans, it uses differentiable rendering to compute gradients directly with respect to mesh vertex positions.

Figure 2: Illustration of our proposed GPJA. With the provided textured template, scans and camera poses, the constraints from HRA L back-propagates derivatives at each iter- ation to guide the warping of the template. The regularized optimization is built on a multiscale scheme with periodic tessellation, and the vertices are updated with a robust modified gradient descent algorithm.

The core architecture consists of:

  1. Holistic Rendering Alignment (HRA): A joint loss function ($\mathcal{L}_{\text{HRA}}$) that combines three distinct cues: color consistency ($\mathcal{L}_{\text{color}}$), depth disparity ($\mathcal{L}_{\text{depth}}$), and surface normal alignment ($\mathcal{L}_{\text{normal}}$).
  2. Mouth Masking Strategy: Interior mouth movements (teeth, tongue) introduce severe occlusion changes. To prevent these from disrupting the optimization, the authors manually create a binary mask on the template texture to exclude the inner mouth socket from the color constraint, ensuring clean and accurate lip contours.
  3. Multiscale Regularized Optimization: Large expression changes involve massive deformations that can cause mesh tearing or self-intersections. GPJA handles this via a coarse-to-fine remeshing scheme paired with a Laplacian-based vertex optimization framework (leveraging the LargeSteps algorithm). This re-parametrizes the vertices to guarantee geometric smoothness during evolution.

Datasets

The method was validated using the FaceScape dataset, a large-scale 3D face benchmark. The authors selected 7 subjects and deliberately picked 10 highly distinct, challenging expressions per subject (featuring extreme deformations and occlusions). For each expression, 6 to 8 multi-view images covering frontal and side views were used as references.

Key Insights and Ideas

  • Differentiable Rendering as a Matchmaker: Instead of trying to calculate explicit 2D-to-3D pixel correspondences—which fail during extreme expressions—automatic differentiation naturally flows gradients from the 2D image pixels back into 3D vertex displacements.
  • The Stabilization Trifecta: Relying solely on color introduces noise in texture-less regions (like cheeks), while relying solely on geometry misses fine-grained features (like eyes). Combining color, depth, and normals creates a mutually stabilizing optimization loop.
  • Laplacian Re-parametrization: Standard gradient descent on dense meshes causes jagged, high-frequency artifacts. By optimizing vertices in a space governed by a discrete Laplacian matrix, the mesh undergoes smooth, large-scale structural changes first, before refining fine details.

Key Results

  • Superior Geometric Accuracy: GPJA consistently outperformed traditional non-rigid ICP (FaceScape TU) and the state-of-the-art deep learning method (NPHM), achieving the lowest L1-Chamfer distance and highest normal consistency across all subjects. For example, on Subject 32, GPJA achieved a Chamfer distance of 0.147 mm, significantly lower than NICP (0.355 mm) and NPHM (0.377 mm).
Geometric error distribution comparison showing that GPJA yields significantly lower errors, particularly around the lips and eyes.
  • Photometric Consistency: Rendered images from the aligned meshes achieved high fidelity, with an average PSNR of 23.88 dB, SSIM of 0.7406, and LPIPS of 0.07469. This matches the visual quality of photo-realistic, per-frame NeRF reconstructions, but with the critical advantage of maintaining a single, production-ready mesh topology and texture map.

Limitations and Future Work

  • Simplified Lighting Model: The framework currently assumes a basic Lambertian reflection model under uniform lighting. This limits its performance when dealing with complex real-world illumination, skin highlights, or specularities.
  • Mouth Interior Modeling: Because features like teeth and the tongue are not explicitly modeled, minor inaccuracies can still occasionally occur along the inner lip boundaries.
  • Computational Efficiency: The optimization process is relatively slow, taking approximately 15 minutes per expression on an NVIDIA RTX 3080 GPU. The authors note that future work will focus on optimizing this efficiency and exploring cross-subject alignment.

Summary updated

Abstract

This paper presents a Geometric-Photometric Joint Alignment~(GPJA) method, which aligns discrete human expressions at pixel-level accuracy by combining geometric and photometric information. Common practices for registering human heads typically involve aligning landmarks with facial template meshes using geometry processing approaches, but often overlook dense pixel-level photometric consistency. This oversight leads to inconsistent texture parametrization across different expressions, hindering the creation of topologically consistent head meshes widely used in movies and games. GPJA overcomes this limitation by leveraging differentiable rendering to align vertices with target expressions, achieving joint alignment in both geometry and photometric appearances automatically, without requiring semantic annotation or pre-aligned meshes for training. It features a holistic rendering alignment mechanism and a multiscale regularized optimization for robust convergence on large deformation. The method utilizes derivatives at vertex positions for supervision and employs a gradient-based algorithm which guarantees smoothness and avoids topological artifacts during the geometry evolution. Experimental results demonstrate faithful alignment under various expressions, surpassing the conventional non-rigid ICP-based methods and the state-of-the-art deep learning based method. In practical, our method generates meshes of the same subject across diverse expressions, all with the same texture parametrization. This consistency benefits face animation, re-parametrization, and other batch operations for face modeling and applications with enhanced efficiency.
BibTeX
@article{wang2025towards,
  author = {Xizhi Wang and Yaxiong Wang and Mengjian Li},
  title = {{Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration}},
  journal = {Comput. Graph.},
  year = {2025},
  doi = {10.1016/j.cag.2025.104214},
}
Figure