TL;DR
The paper introduces Geometric-Photometric Joint Alignment (GPJA), a novel method for tracking and registering 3D facial expressions with pixel-level accuracy. By leveraging differentiable rendering to combine 3D geometric data with dense 2D photometric appearance, GPJA automatically produces topologically consistent meshes with a shared texture parametrization across diverse expressions without requiring manual annotations or extensive training data.
What Problem Does This Paper Address?
Creating high-fidelity, topologically consistent 3D head meshes across different facial expressions is a foundational requirement for modern movies, video games, and VR applications. Traditional registration pipelines typically rely on non-rigid Iterative Closest Point (ICP) algorithms to warp a generic template mesh onto a 3D scan using sparse landmarks or 2D optical flow.
However, these approaches often ignore dense, pixel-level photometric (color) consistency. This oversight leads to texture misalignment—such as a mole, wrinkle, or lip contour shifting unnaturally across different expressions—which requires tedious, manual frame-by-frame clean-up. While recent deep learning methods attempt to automate this, they often struggle to capture fine-grained details and require massive, pre-aligned datasets for training.
GPJA aims to solve these challenges by jointly optimizing both geometry and appearance automatically, ensuring that the same physical point on a subject's face maps to the exact same texture coordinate regardless of the expression.
Methods
GPJA avoids explicit, error-prone correspondence matching by treating registration as an inverse rendering problem. Guided by multi-view images and 3D scans, it uses differentiable rendering to compute gradients directly with respect to mesh vertex positions.
The core architecture consists of:
- Holistic Rendering Alignment (HRA): A joint loss function ($\mathcal{L}_{\text{HRA}}$) that combines three distinct cues: color consistency ($\mathcal{L}_{\text{color}}$), depth disparity ($\mathcal{L}_{\text{depth}}$), and surface normal alignment ($\mathcal{L}_{\text{normal}}$).
- Mouth Masking Strategy: Interior mouth movements (teeth, tongue) introduce severe occlusion changes. To prevent these from disrupting the optimization, the authors manually create a binary mask on the template texture to exclude the inner mouth socket from the color constraint, ensuring clean and accurate lip contours.
- Multiscale Regularized Optimization: Large expression changes involve massive deformations that can cause mesh tearing or self-intersections. GPJA handles this via a coarse-to-fine remeshing scheme paired with a Laplacian-based vertex optimization framework (leveraging the LargeSteps algorithm). This re-parametrizes the vertices to guarantee geometric smoothness during evolution.
Datasets
The method was validated using the FaceScape dataset, a large-scale 3D face benchmark. The authors selected 7 subjects and deliberately picked 10 highly distinct, challenging expressions per subject (featuring extreme deformations and occlusions). For each expression, 6 to 8 multi-view images covering frontal and side views were used as references.
Key Insights and Ideas
- Differentiable Rendering as a Matchmaker: Instead of trying to calculate explicit 2D-to-3D pixel correspondences—which fail during extreme expressions—automatic differentiation naturally flows gradients from the 2D image pixels back into 3D vertex displacements.
- The Stabilization Trifecta: Relying solely on color introduces noise in texture-less regions (like cheeks), while relying solely on geometry misses fine-grained features (like eyes). Combining color, depth, and normals creates a mutually stabilizing optimization loop.
- Laplacian Re-parametrization: Standard gradient descent on dense meshes causes jagged, high-frequency artifacts. By optimizing vertices in a space governed by a discrete Laplacian matrix, the mesh undergoes smooth, large-scale structural changes first, before refining fine details.
Key Results
- Superior Geometric Accuracy: GPJA consistently outperformed traditional non-rigid ICP (FaceScape TU) and the state-of-the-art deep learning method (NPHM), achieving the lowest L1-Chamfer distance and highest normal consistency across all subjects. For example, on Subject 32, GPJA achieved a Chamfer distance of 0.147 mm, significantly lower than NICP (0.355 mm) and NPHM (0.377 mm).
- Photometric Consistency: Rendered images from the aligned meshes achieved high fidelity, with an average PSNR of 23.88 dB, SSIM of 0.7406, and LPIPS of 0.07469. This matches the visual quality of photo-realistic, per-frame NeRF reconstructions, but with the critical advantage of maintaining a single, production-ready mesh topology and texture map.
Limitations and Future Work
- Simplified Lighting Model: The framework currently assumes a basic Lambertian reflection model under uniform lighting. This limits its performance when dealing with complex real-world illumination, skin highlights, or specularities.
- Mouth Interior Modeling: Because features like teeth and the tongue are not explicitly modeled, minor inaccuracies can still occasionally occur along the inner lip boundaries.
- Computational Efficiency: The optimization process is relatively slow, taking approximately 15 minutes per expression on an NVIDIA RTX 3080 GPU. The authors note that future work will focus on optimizing this efficiency and exploring cross-subject alignment.
Summary updated