Adaptive 3D Face Reconstruction from a Single Image

Kun Li, Jing Yang, Nianhong Jiao, Jinsong Zhang, Yu-Kun LaiarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel method for reconstructing 3D face models from a single 2D image, even in challenging scenarios like extreme poses and partial occlusions. By jointly optimizing 2D and 3D landmark information and adaptively weighting their contributions, the method achieves more accurate and robust face reconstructions than previous approaches.

What Problem Does This Paper Address?

Reconstructing a 3D face from a single 2D image is a long-standing challenge in computer vision. Existing methods often struggle with variations in head pose (e.g., looking sideways) and situations where parts of the face are hidden (occlusions). These issues lead to inaccurate 3D models because the precise 2D locations of facial features (landmarks) become uncertain, and the lack of depth information makes it hard to infer the hidden parts of the face. This paper aims to overcome these limitations by developing a more robust and accurate 3D face reconstruction technique.

Methods

The core of this work lies in a two-pronged approach:

  • Coarse-to-Fine Pose Estimation: Instead of directly estimating the complex 3D pose of the face, the method first estimates a coarse pose using both 2D and 3D landmarks. This initial estimate is then refined through a more detailed process that leverages the 3D information to better handle occluded landmarks. This iterative refinement helps to resolve ambiguities and improve pose accuracy.
  • Joint 2D and 3D Optimization with Adaptive Re-weighting: The paper proposes a unified framework that simultaneously optimizes 2D and 3D landmark information. Crucially, it introduces an "adaptive re-weighting" mechanism. This means the method dynamically adjusts how much it trusts the 2D versus the 3D landmark information based on the estimated pose. For instance, in extreme poses where 2D landmarks might be less reliable, more weight is given to the 3D depth information. Conversely, for near-frontal poses, 2D landmarks are given higher importance. This adaptive weighting allows the model to leverage the most reliable information available for each specific scenario.

Datasets

The authors evaluated their method on the following datasets:

  • AFLW2000: A large-scale face database known for its diverse poses and perspectives.
  • MICC: A dataset comprising 53 videos with varying resolutions and conditions, used for evaluating performance on partial occlusions and extreme poses.
  • Indoor-Cooperative videos: Used for ablation studies, specifically selecting non-frontal images to demonstrate the effectiveness of individual components.

Key Insights and Ideas

  • 3D Depth is Crucial for Occlusions and Extreme Poses: The paper strongly emphasizes that relying solely on 2D landmarks is insufficient for accurate reconstruction when faces are heavily occluded or in extreme poses. The inherent depth information from 3D landmarks provides vital constraints that are otherwise missing.
  • Adaptive Weighting is Key to Robustness: The adaptive re-weighting of 2D and 3D landmarks is a significant contribution. It allows the system to intelligently balance the reliability of different cues based on the pose, leading to more consistent and accurate results across a wider range of challenging conditions.
  • Coarse-to-Fine Pose Estimation Simplifies a Complex Problem: By breaking down pose estimation into an initial coarse step followed by a refined step, the method tackles the inherent difficulty of directly predicting accurate 3D poses from a single image.

Key Results

  • Improved Accuracy: The proposed method achieves state-of-the-art results on benchmark datasets like AFLW2000 and MICC, outperforming existing methods, particularly for non-frontal images and those with occlusions. For example, on the MICC dataset, their method (2D+3D+P+W) achieved a 3DRMSE of 1.812 for left views and 1.835 for right views, significantly lower than other variants.
  • Robustness: Qualitative results demonstrate the method's ability to generate high-quality 3D face reconstructions from images with hair occlusions, small pose rotations, and even significant occlusions like glasses.
  • Automatic and Unsupervised: The method is fully automatic and does not require manual intervention, making it more practical for real-world applications.

Limitations and Future Work

  • Landmark Detection Accuracy: The paper acknowledges that the accuracy of the final reconstruction is still influenced by the initial accuracy of 2D and 3D landmark detection. Errors in landmark detection, especially in cases of severe occlusion, can lead to reconstruction failures (as shown in Figure 9).
  • Future Directions: The authors plan to improve landmark detection accuracy by iteratively refining the face reconstruction. They also aim to explore combining their optimization method with learning-based priors or representations to further enhance performance.

Summary updated

Abstract

3D face reconstruction from a single image is a challenging problem, especially under partial occlusions and extreme poses. This is because the uncertainty of the estimated 2D landmarks will affect the quality of face reconstruction. In this paper, we propose a novel joint 2D and 3D optimization method to adaptively reconstruct 3D face shapes from a single image, which combines the depths of 3D landmarks to solve the uncertain detections of invisible landmarks. The strategy of our method involves two aspects: a coarse-to-fine pose estimation using both 2D and 3D landmarks, and an adaptive 2D and 3D re-weighting based on the refined pose parameter to recover accurate 3D faces. Experimental results on multiple datasets demonstrate that our method can generate high-quality reconstruction from a single color image and is robust for self-occlusion and large poses.
BibTeX
@misc{li2020adaptive,
  author = {Kun Li and Jing Yang and Nianhong Jiao and Jinsong Zhang and Yu-Kun Lai},
  title = {{Adaptive 3D Face Reconstruction from a Single Image}},
  year = {2020},
  eprint = {2007.03979},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.03979},
}
Figure