Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions

Egor Burkov, Ruslan Rakhimov, Aleksandr Safin, Evgeny Burnaev, Victor Lempitsky

Abstract

We present an approach for the reconstruction of textured 3D meshes of human heads from one or few views. Since such few-shot reconstruction is underconstrained, it requires prior knowledge which is hard to impose on traditional 3D reconstruction algorithms. In this work, we rely on the recently introduced 3D representation – neural implicit functions – which, being based on neural networks, allows to naturally learn priors about human heads from data, and is directly convertible to textured mesh. Namely, we extend NeuS, a state-of-the-art neural implicit function formulation, to represent multiple objects of a class (human heads in our case) simultaneously. Our model is trained on just a hundred smartphone videos and does not require any scanned 3D data. Afterwards, the model can fit novel heads in the few-shot or one-shot modes with good results.

Introduction

Creating a high-fidelity 3D model of a human head from a single "in-the-wild" photograph is a significant challenge in computer vision. This "one-shot" reconstruction is inherently underconstrained because the input lacks information about occluded areas, such as the back of the head. To solve this, systems must rely on strong prior knowledge.

An 'in-the-wild' input image from a historical painting used for one-shot reconstruction.

Traditional approaches often utilize 3D Morphable Models (3DMMs), which provide coarse shapes but struggle with fine details like hair. Conversely, modern deep learning methods typically require massive datasets of 3D scans—which are difficult and expensive to acquire—or synthetic data that often lacks realism. Multi-NeuS bridges this gap by learning high-fidelity 3D priors directly from easily obtainable 2D video collections without requiring any scanned 3D data.

Methodology

Multi-NeuS is built upon NeuS, a method that represents 3D surfaces as a Signed Distance Function (SDF). While the original NeuS is designed for single-scene reconstruction from many views, Multi-NeuS extends this to handle multiple identities simultaneously through several key innovations:

  • Hybrid Architecture: The neural network features shared layers that learn the general structure of human heads and scene-specific layers that capture individual identity.
  • Low-Rank Regularization: Identity-specific weights are represented as a linear combination of a small set of basis vectors. This compact representation acts as a powerful regularizer, preventing overfitting and ensuring realistic geometry even with minimal input.
  • Two-Stage Optimization: The model first undergoes meta-learning on a multi-identity dataset to learn an "average" head model. When presented with a new subject, it performs a coarse fit of identity layers followed by a fine-tuning of the entire network.
  • Camera Refinement: To handle "in-the-wild" photos without known camera parameters, the model jointly optimizes the estimated camera position and focal length during the fitting process.

Datasets and Evaluation

The model was trained on the SmartPortraits dataset, which consists of short smartphone videos of 107 individuals. Notably, this dataset contains only RGB video and no 3D geometry. For quantitative evaluation, the researchers used the H3DS dataset, which provides high-quality 3D ground-truth scans for 10 individuals, allowing for precise measurement of reconstruction error in millimeters.

Key Results

Multi-NeuS demonstrates remarkable data efficiency and accuracy. On the H3DS benchmark, it achieved a facial reconstruction error of 1.84mm, matching the performance of H3D-Net, despite the latter being trained on a dataset of 10,000 3D scans.

3D reconstruction results showing multiple views and normal maps derived from a single input image.

The model's robustness is further evidenced by its success on non-photographic data, such as historical paintings and artistic portraits. It maintains the subject's identity while producing plausible, "watertight" 3D meshes suitable for graphics applications.

Limitations and Future Work

Despite its successes, Multi-NeuS has limitations:

  • Occluded Regions: Since training videos primarily cover the front of the face (±45°), the model can produce artifacts or hollow regions in the back-of-the-head areas.
  • Diversity: The current training set has limited variation in hairstyles, skin tones, and accessories.
  • Static Geometry: The model currently only handles neutral expressions and does not yet support rigging for animation.

Future work may involve incorporating larger, more diverse datasets and exploring "learned gradient descent" to further enhance generalization across all human types.

Summary updated

Abstract

We present an approach for the reconstruction of textured 3D meshes of human heads from one or few views. Since such few-shot reconstruction is underconstrained, it requires prior knowledge which is hard to impose on traditional 3D reconstruction algorithms. In this work, we rely on the recently introduced 3D representation $\unicode{x2013}$ neural implicit functions $\unicode{x2013}$ which, being based on neural networks, allows to naturally learn priors about human heads from data, and is directly convertible to textured mesh. Namely, we extend NeuS, a state-of-the-art neural implicit function formulation, to represent multiple objects of a class (human heads in our case) simultaneously. The underlying neural net architecture is designed to learn the commonalities among these objects and to generalize to unseen ones. Our model is trained on just a hundred smartphone videos and does not require any scanned 3D data. Afterwards, the model can fit novel heads in the few-shot or one-shot modes with good results.
BibTeX
@inproceedings{burkov2023multi,
  author = {Egor Burkov and Ruslan Rakhimov and Aleksandr Safin and Evgeny Burnaev and Victor Lempitsky},
  title = {{Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=MwJGhVK0zc},
}
Figure