Abstract
Three-dimensional face dense alignment and reconstruction in the wild is a challenging problem as partial facial information is commonly missing in occluded and large pose face images. Large head pose variations also increase the solution space and make the modeling more difficult. Our key idea is to model occlusion and pose to decompose this challenging task into several relatively more manageable subtasks. To this end, we propose an end-to-end framework, termed as Self-aligned Dual face Regression Network (SADRNet), which predicts a pose-dependent face and a pose-independent face. They are combined by an occlusion-aware self-alignment to generate the final 3D face. Extensive experiments on two popular benchmarks, AFLW2000-3D and Florence, demonstrate that the proposed method achieves significant superior performance over existing state-of-the-art methods.
Introduction
The task of 3D Dense Face Alignment and Face Reconstruction (3D-DFAFR) from a single 2D image is critical for various computer vision applications. However, "in-the-wild" scenarios present significant hurdles, primarily due to occlusions (e.g., hair, glasses, hands) and large head poses. These factors lead to missing data and an expanded geometric search space, making it difficult for standard models to distinguish between intrinsic shape variations and perspective distortions.
Existing approaches generally fall into two categories: model-based methods using 3D Morphable Models (3DMMs), which can be too rigid for fine details, and model-free methods that directly regress coordinates. Model-free methods often struggle with occlusions as they lack a geometric prior of facial structure. SADRNet addresses these issues by decomposing the task into manageable subtasks through a dual-regression approach.
Methodology
SADRNet utilizes an encoder-decoder architecture designed to disentangle pose and shape while remaining robust to occlusions. The framework consists of several novel components:
Dual Regression Heads
The decoder predicts two distinct UV position maps:
- Pose-dependent face (P): A direct regression of the face as it appears in the input image.
- Shape Deformation (D): Rather than predicting a pose-independent face (S) from scratch, the model predicts the deformation from a standard mean face template (S̄). This simplifies the learning process by providing a strong geometric prior.
Occlusion-Aware Attention
A side-branch in the encoder generates an attention mask (A) that highlights visible facial regions. By suppressing features from occluded areas, the network prevents "garbage" data from corrupting the final 3D reconstruction.
Self-Alignment Module
Instead of regressing unstable pose parameters like Euler angles, SADRNet uses the predicted visibility mask to select reliable landmarks. It then calculates an optimal similarity transformation (scale, rotation, and translation) to align the pose-independent shape (S) with the pose-dependent face (P). This closed-form alignment is more robust than direct regression and allows for error correction.
Experiments and Results
The model was trained on the 300W-LP dataset, augmented with synthetic occlusions and extreme yaw angles (up to 105°) to improve profile-view performance. Evaluation was conducted on the AFLW2000-3D and Florence benchmarks.
Key Findings
- State-of-the-Art Accuracy: SADRNet outperformed existing methods like PRN and 3DDFAv2 on AFLW2000-3D. On the Florence dataset, it achieved a 13% improvement in Normalized Mean Error (NME) over previous top-tier methods.
- Efficiency and Compactness: The model runs at 70 FPS on a single GTX 1080 Ti, making it suitable for real-time use. Its size is only 60MB, significantly smaller than many competitors (e.g., VRN at 1.5GB).
- Robustness: The use of a mean face template ensures that the model can infer hidden geometry even when large portions of the face are occluded.
Conclusion
SADRNet demonstrates that disentangling pose and shape through dual regression, combined with an occlusion-aware self-alignment mechanism, significantly improves 3D face reconstruction in challenging environments. Future work will focus on reconstructing facial textures and moving toward weakly-supervised learning to reduce the reliance on fully-annotated 3D mesh data.
Summary updated