Abstract
Monocular face reconstruction is a challenging task in computer vision, which aims to recover 3D face geometry from a single RGB face image. Recently, deep learning based methods have achieved great improvements on monocular face reconstruction. However, for deep learning-based methods to reach optimal performance, it is paramount to have large-scale training images with ground-truth 3D face geometry, which is generally difficult for human to annotate. To tackle this problem, we propose a semi-supervised monocular reconstruction method, which jointly optimizes a shape-preserved domain-transfer CycleGAN and a shape estimation network. The framework is semi-supervised trained with 3D rendered images with ground-truth shapes and in-the-wild face images without any extra annotation. The CycleGAN network transforms all realistic images to have the rendered style and is end-to-end trained within the overall framework. This is the key difference compared with existing CycleGAN-based learning methods, which just used CycleGAN as a separate training sample generator. Novel landmark consistency loss and edge-aware shape estimation loss are proposed for our two networks to jointly solve the challenging face reconstruction problem. Extensive experiments on public face reconstruction datasets demonstrate the effectiveness of our overall method as well as the individual components.
Introduction
Monocular 3D face reconstruction—recovering a 3D model from a single 2D image—is a cornerstone of computer vision. While deep learning has pushed the field forward, it faces a significant data bottleneck. Synthetic data provides perfect 3D ground truth but often lacks realism, leading to models that fail on real-world photos. Conversely, real-world data is abundant, but obtaining accurate 3D ground truth for "in-the-wild" images (with varying lighting, occlusions, and poses) is nearly impossible for manual annotation.
This paper introduces a semi-supervised framework that bridges this "domain gap." By training a domain-transfer CycleGAN and a shape estimation network end-to-end, the authors leverage perfectly labeled synthetic data while maintaining high accuracy on real-world images, particularly in extreme poses.
Methodology
The proposed system is trained in a semi-supervised, end-to-end fashion, consisting of two primary modules: a Shape-Preserved Domain Transfer network and a Shape Estimation Network.
Shape-Preserved Domain Transfer (CycleGAN)
Instead of attempting to make synthetic images look real, which is difficult due to complex backgrounds and facial clutter (hair, glasses, beards), the authors do the opposite. They train a generator to transform realistic "in-the-wild" images into a simplified "rendered style." This simplification strips away non-geometric details, allowing the shape estimator to focus on the underlying facial structure.
Shape Estimation Network
This network takes the "rendered-style" image and predicts a 3D shape in the form of a UV position map. Unlike previous works that treat CycleGAN as a separate pre-processing step, this framework integrates both networks into a single optimization loop. This ensures the GAN learns to produce images specifically optimized for the shape estimator.
Key Losses
- Landmark Consistency Loss: To prevent the GAN from warping facial geometry during style transfer, a pre-trained 2D landmark detector is used. The model enforces that the landmarks of the original real image match those of the transformed "rendered" image.
- Edge-aware Shape Estimation Loss: For real images lacking 3D ground truth, the model projects its estimated 3D shape back onto a 2D plane. It then calculates the distance between the edges of this projection and the edges of the original face (eyes, nose, mouth, chin), providing a strong unsupervised signal for shape accuracy.
Experiments and Results
The model was trained using 100,000 synthetic images generated from the Basel Face Model (BFM) and unlabeled real-world images. Evaluation was performed on public datasets including AFLW2000-3D, AFLW-LFPA (containing large-angle views), and Florence (high-accuracy 3D scans).
Key Insights
- Simplification over Complication: It is computationally more efficient to strip away clutter from a real image than to synthesize realistic clutter onto a 3D model.
- Robustness to Occlusion: Because the GAN maps real images to a "clean" rendered style, it effectively "hallucinates" parts of the face hidden by sunglasses or microphones, enabling the shape estimator to work on a complete facial structure.
Quantitative and Qualitative Performance
The method outperformed state-of-the-art supervised methods on images with extreme yaw angles (60°–90°). On the AFLW-LFPA dataset, the model achieved the smallest average Normalized Mean Error ($NME_{3d} = 3.703$), despite not using any 3D ground truth from that dataset during training. Visual results confirm the model's success in removing beards and sunglasses during domain transfer, leading to more accurate 3D reconstructions than traditional supervised models like 3DDFA or PRNet.
Conclusion
This research demonstrates that an end-to-end, semi-supervised approach can effectively overcome the data scarcity in 3D face reconstruction. By simplifying the domain transfer task and anchoring it with geometric constraints like landmarks and edges, the model achieves high accuracy on challenging "in-the-wild" data. Future work may address the recovery of high-fidelity facial textures and skin details, as well as ensuring temporal consistency for video-based reconstruction.
Summary updated