Learning Neural Proto-Face Field for Disentangled 3D Face Modeling in the Wild

Zhenyu Zhang, Renwang Chen, Weijian Cao, Ying Tai, Chengjie Wang

Introduction

Reconstructing high-fidelity 3D faces from unconstrained, "in-the-wild" photo collections is a fundamental challenge in computer vision. The problem is inherently ill-posed, as a single 2D image can correspond to numerous 3D shapes. While traditional 3D Morphable Models (3DMMs) provide a robust statistical prior, they often fail to capture the unique, fine-grained details of an individual's identity. Conversely, modern neural rendering techniques like Neural Radiance Fields (NeRF) offer high detail but are notoriously fragile when faced with the noise, occlusions, and extreme lighting typical of real-world photography.

The Neural Proto-face Field (NPF) framework addresses these limitations by leveraging multi-image priors. By aggregating information from a collection of photos of the same person, NPF constructs a robust "prototype" face that disentangles core identity from transient expressions and environmental artifacts.

The NPF Framework

NPF introduces a multi-stage pipeline designed to move beyond the limitations of single-view reconstruction. The core of the system is built upon EG3D, a state-of-the-art 3D-aware generative model utilizing a tri-plane representation for efficient, high-quality rendering.

1. Identity-aware Aggregation

Rather than treating all images in a collection as equally valid, NPF employs an uncertainty encoder. This component predicts the reliability of style codes extracted from each image. By performing a weighted average based on these uncertainty scores, the model aggregates features into a single pivotal proto code ($\bar{s}$). This code serves as the anchor for the subject's core identity, effectively filtering out "liars"—pixels affected by harsh shadows or occlusions.

2. Disentangled Deformation Modeling

To account for facial movements across a photo collection, NPF separates identity from expression. It utilizes 3DMM expression parameters and global image features to learn a deformation field. This allows the static "prototype" to be realistically morphed into various expressions without compromising the underlying 3D geometry.

NPF handles complex expressions by disentangling identity from deformation.

3. Robust Two-Stage Fitting

The final reconstruction is achieved through a two-step optimization process:

  • Multi-image Warm-up: The model is first tuned against the entire collection using a geometry consistency loss, ensuring a stable 3D shape from multiple viewpoints.
  • Target-image Fitting: The model is then refined to match a specific target image. This stage recovers fine-grained details like skin pores and wrinkles while utilizing a representation consistency loss to prevent geometric collapse.

Key Insights

  • Uncertainty as a Filter: Modeling uncertainty allows the network to ignore transient artifacts (like hair or shadows) and focus on consistent 3D structures.
  • The Prototype Anchor: Establishing a 3D-consistent prototype first mitigates "monocular ambiguity," where a single-view model might misinterpret a shadow as a geometric feature.
  • Disentanglement for Robustness: Separating identity, expression, and scene-specific details enables more robust editing and animation compared to entangled neural renderers.
NPF captures high-fidelity identity details while maintaining geometric consistency.

Results and Evaluation

NPF demonstrates superior performance across several benchmarks:

  • Geometry Accuracy: On the MICC dataset, NPF achieved a point-to-plane (p2p) distance of 1.77mm, significantly outperforming non-parametric methods like LAP (2.32mm) and Unsup3D (2.48mm).
  • Surface Detail: In facial normal evaluations on the Photoface dataset, NPF reached a Mean Angle Deviation (MAD) of 11.87°, the lowest among tested methods, indicating sharper and more accurate surface reconstruction.
  • Robustness: Qualitatively, NPF maintains plausible face shapes even under extreme rotations (up to ±60°), where baseline models often produce distorted or "flat" results.

Conclusion

Neural Proto-face Field represents a significant step forward in robust 3D face modeling. By moving from single-image to multi-image priors and incorporating uncertainty modeling, NPF achieves photo-realistic textures and precise geometry in unconstrained environments. While currently requiring multiple images of a subject, future work aims to further disentangle intrinsic factors like albedo and lighting to enhance control and potentially reduce the data requirements for high-quality reconstruction.

Summary updated

Abstract

Generative models show good potential for recovering 3D faces beyond limited shape assumptions. While plausible details and resolutions are achieved, these models easily fail under extreme conditions of pose, shadow or appearance, due to the entangled fitting or lack of multi-view priors. To address this problem, this paper presents a novel Neural Proto-face Field (NPF) for unsupervised robust 3D face modeling. Instead of using constrained images as Neural Radiance Field (NeRF), NPF disentangles the common/specific facial cues, i.e., ID, expression and scene-specific details from in-the-wild photo collections. Specifically, NPF learns a face prototype to aggregate 3D-consistent identity via uncertainty modeling, extracting multi-image priors from a photo collection. NPF then learns to deform the prototype with the appropriate facial expressions, constrained by a loss of expression consistency and personal idiosyncrasies. Finally, NPF is optimized to fit a target image in the collection, recovering specific details of appearance and geometry. In this way, the generative model benefits from multi-image priors and meaningful facial structures. Extensive experiments on benchmarks show that NPF recovers superior or competitive facial shapes and textures, compared to state-of-the-art methods.
BibTeX
@inproceedings{zhang2023learning,
  author = {Zhenyu Zhang and Renwang Chen and Weijian Cao and Ying Tai and Chengjie Wang},
  title = {{Learning Neural Proto-Face Field for Disentangled 3D Face Modeling in the Wild}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00045},
}
Figure