TL;DR
This paper introduces a novel method for reconstructing 3D face models from single images, specifically addressing the challenge of occlusions. By leveraging generative face parsing maps guided by landmarks, their approach effectively reconstructs complete facial textures even when parts of the face are hidden, outperforming existing methods in both accuracy and robustness.
What Problem Does This Paper Address?
Reconstructing a 3D face model from a single 2D image is a well-studied problem with applications in areas like facial recognition and digital entertainment. However, existing methods typically struggle when the input image contains occlusions, such as glasses, jewelry, or hair. These occlusions can lead to incomplete or inaccurate 3D reconstructions, limiting the practical applicability of these techniques in real-world scenarios. The paper aims to bridge this gap by developing a system that can generate convincing 3D face textures even when faced with these occluded inputs.
Methods
The core of the proposed method lies in a two-stage approach:
- Landmark Prediction and Face Parsing Map Generation: The system first predicts 68 facial landmarks from the input image. These landmarks are then used to guide the generation of a complete face parsing map. This map is crucial for identifying and "filling in" the occluded regions. The authors propose a novel algorithm that combines two parsing maps: one generated by an encoder-decoder network (Na) and another derived from the predicted landmarks (Mβ). This combination (My) effectively creates a parsing map that represents the complete facial features, even those that were originally occluded.
- Generative Face Image Synthesis: Using the complete face parsing map (My) and the original input image (Ifac), a Generative Adversarial Network (GAN) based Face Image Synthesis Network (FISN) is employed. This network, built upon the Pix2PixHD architecture, synthesizes a complete face image that can then be used for 3D reconstruction. The FISN utilizes a Spatial Feature Transform (SFT) layer to condition the synthesis on the parsing map and employs semantic region-adaptive normalization (SEAN) to preserve semantic information.
The 3D reconstruction itself is then achieved by regressing coefficients for a 3D Morphable Model (3DMM) using a ResNet-50 network. The loss function for this reconstruction incorporates multiple components: landmark loss, pixel-wise loss, regularization loss, and a face feature level loss (using FaceNet for recognition-level comparison).
Datasets
- Training: The paper mentions using the CelebAMask-HQ dataset for training the face parsing model (Na) and the FISN.
- Evaluation: The Labeled Faces in the Wild (LFW) dataset is used for evaluating the robustness of the reconstructed shapes to occlusions. The MICC Florence dataset is also used for quantitative comparison.
Key Insights and Ideas
- Parsing Maps as a Bridge for Occlusions: The central idea is that a complete face parsing map, even if generated from an occluded input, can provide the necessary semantic information to guide the reconstruction of missing facial regions.
- Landmark-Guided Parsing: Using predicted landmarks to construct a part of the parsing map is a clever way to ensure the topological correctness of facial features, especially in occluded areas.
- Synthesizing Complete Faces, Then Reconstructing: Instead of directly reconstructing the 3D model from an occluded image, the approach first synthesizes a complete 2D face image and then uses that for 3D reconstruction. This decouples the challenges of occlusion handling and 3D reconstruction.
- Robustness to Occlusions: The method demonstrates a significant improvement in handling various types of occlusions, a common failure point for existing 3D face reconstruction techniques.
Key Results
- Qualitative: The paper presents compelling qualitative results showing that their method can reconstruct complete and realistic 3D face models even with significant occlusions (glasses, jewelry, hair). Visual comparisons with baseline methods like 3DDFA, DF2Net, and Chen et al. highlight the superiority of their approach in handling these challenging scenarios.
- Quantitative:
- On the MICC Florence dataset, the method shows a significantly lower average 90% largest error compared to other methods when handling occlusions.
- On the LFW benchmark, the method demonstrates robustness to occlusions, with only a limited drop in performance (e.g., 100%-EER, Accuracy, nAUC) when tested with occluded faces compared to unoccluded ones.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while their method is robust, occlusions do impact recognition performance, as seen in the LFW results.
- The 3DMM itself limits the fine details of shape, and the method relies on adding refined texture separately.
- Future Work: The authors suggest exploring the integration of Transformer architectures to further enhance accuracy.
Summary updated