DF2Net: A Dense-Fine-Finer Network for Detailed 3D Face Reconstruction

Xiaoxing Zeng, Xiaojiang Peng, Yu Qiao

Introduction

Reconstructing high-fidelity 3D face geometry from a single 2D image remains a fundamental challenge in computer vision. While traditional methods like 3D Morphable Models (3DMM) provide a solid foundation, they often result in "over-smooth" reconstructions that lack the subtle micro-geometries—such as forehead wrinkles and crow’s feet—that define individual character and realism. This ICCV paper introduces DF2Net, a novel deep learning framework designed to bridge the gap between coarse global shape and fine-grained local detail.

The DF2Net Architecture

The core innovation of DF2Net is its three-stage cascaded architecture, which progressively refines facial geometry from a "Dense" base to "Fine" and "Finer" details.

1. D-Net (Dense Stage)

The first stage, D-Net, utilizes a U-Net architecture to map the input image directly to a dense depth map. Unlike previous approaches that regress low-dimensional 3DMM parameters, D-Net learns to recover dense depth information, providing a more flexible starting point for detailed reconstruction.

2. F-Net (Fine Stage)

F-Net refines the initial depth map by fusing features from both the original RGB image and the D-Net output. This stage employs a "hypercolumn" architecture and is trained using a Shape-from-Shading (SfS) loss. By leveraging lighting and shading cues, F-Net begins to recover more nuanced geometric features that are often lost in the initial dense mapping.

3. Fr-Net (Finer Stage)

The final stage, Fr-Net, focuses exclusively on high-frequency micro-details. It integrates the original color image at various scales through multi-resolution hypercolumn blocks, maintaining a "spatial memory" of fine details. Instead of predicting a full face, Fr-Net predicts a residual displacement map. This allows the network to specialize in learning the tiny corrections—like pores and sharp wrinkles—needed to enhance the existing model.

Input image featuring significant facial detail and wrinkles.

Training and Datasets

To achieve such high levels of detail, the authors curated a diverse training pipeline involving three types of data:

  • Synthetic Data: 60,000 image-depth pairs generated via 3DMM with varying poses and lighting.
  • Pseudo-Ground Truth: 10,000 pairs generated by running existing state-of-the-art algorithms (like PRNet) on real images.
  • High-Detail Real Images: 5,000 high-quality images from the CACD dataset, specifically chosen for their rich facial textures (e.g., elderly faces with prominent wrinkles).

This hybrid training strategy allows the model to learn robust structural information from synthetic data while capturing the realism of real-world facial features through residual learning.

Results and Performance

DF2Net demonstrates superior performance in both quantitative benchmarks and qualitative visual assessments. On the BU-3DFE dataset, a standard for 3D face reconstruction, the model achieved a Mean Error of 3.37, outperforming established methods like Pix2vertex (3.51) and HPEN (3.85).

Qualitatively, the difference is striking. While competing methods often produce bumpy artifacts or erase fine lines, DF2Net successfully recovers sharp, anatomically correct details like forehead furrows and crow's feet, as seen in the reconstructed mesh below.

The reconstructed 3D mesh demonstrating the recovery of fine-grained wrinkles and geometry.

Conclusion

DF2Net represents a significant step forward in single-image 3D face reconstruction. By decomposing the problem into hierarchical stages and utilizing residual learning for micro-details, it achieves a level of realism previously difficult to reach. While the computational cost of the Shape-from-Shading process remains a challenge, the framework's ability to maintain spatial memory through multi-resolution fusion sets a new standard for capturing the human face in 3D.

Summary updated

Abstract

Reconstructing the detailed geometric structure from a single face image is a challenging problem due to its ill-posed nature and the fine 3D structures to be recovered. This paper proposes a deep Dense-Fine-Finer Network (DF2Net) to address this challenging problem. DF2Net decomposes the reconstruction process into three stages, each of which is processed by an elaborately-designed network, namely D-Net, F-Net, and Fr-Net. D-Net exploits a U-net architecture to map the input image to a dense depth image. F-Net refines the output of D-Net by integrating features from depth and RGB domains, whose output is further enhanced by Fr-Net with a novel multi-resolution hypercolumn architecture. In addition, we introduce three types of data to train these networks, including 3D model synthetic data, 2D image reconstructed data, and fine facial images. We elaborately exploit different datasets (or combination) together with well-designed losses to train different networks. Qualitative evaluation indicates that our DF2Net can effectively reconstruct subtle facial details such as small crow's feet and wrinkles. Our DF2Net achieves performance superior or comparable to state-of-the-art algorithms in qualitative and quantitative analyses on real-world images and the BU-3DFE dataset. Code and the collected 70K image-depth data will be publicly available.
BibTeX
@inproceedings{zeng2019df,
  author = {Xiaoxing Zeng and Xiaojiang Peng and Yu Qiao},
  title = {{DF2Net: A Dense-Fine-Finer Network for Detailed 3D Face Reconstruction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00240},
}
Figure