MoFaNeRF: Morphable Facial Neural Radiance Field

Yiyu Zhuang, Hao Zhu, Xusen Sun, Xun Cao

TL;DR

MoFaNeRF is the first facial parametric model built entirely upon a Neural Radiance Field (NeRF) that supports image fitting, random generation, and interactive manipulation. By disentangling facial shape, expression, and appearance into a unified vector space, it achieves photo-realistic synthesis of complex details like pupils and beards that traditional mesh-based models struggle to capture.

What Problem Does This Paper Address?

For decades, the 3D Morphable Model (3DMM) has been the gold standard for modeling human faces. It allows researchers to represent faces using a compact set of parameters for shape and texture, making it easy to "rig" a face into new expressions or "fit" it to a single photo. However, 3DMMs rely on mesh-based rendering, which often produces "uncanny" results. They struggle with non-Lambertian surfaces (like the wetness of eyes) and fine-grained geometry (like individual beard hairs or the inside of a mouth).

On the other hand, Neural Radiance Fields (NeRF) have revolutionized photo-realistic view synthesis. While NeRFs produce stunningly realistic images, they are typically "baked" into a specific scene or person. Making a NeRF "morphable"—allowing it to represent thousands of different people and expressions in a single, editable model—is a significant technical challenge. MoFaNeRF bridges this gap, combining the editability of 3DMMs with the rendering power of NeRF.

Methods

The core of MoFaNeRF is a parametric mapping function that takes five inputs: a 3D position ($x$), a viewing direction ($d$), and three disentangled codes for shape ($\beta$), appearance ($\alpha$), and expression ($\epsilon$).

Fig. 2. MoFaNeRF takes appearance code α, shape code β, expression code ϵ, posi- tion code x and view direction d as input, synthesizing a coarse result which is then refined by a RefineNet. As shown in the right bottom corner, MoFaNeRF can be used in generating (synthesize free-view images given parameters) or fitting (optimize for parameters given a single image).

Key technical innovations include:

  • Identity-Specific Modulation (ISM): The authors found that standard MLPs tend to "average out" expressions across different people. ISM uses the shape code to modulate the expression code, capturing the unique way an individual's face moves.
  • Texture Encoding Module (TEM): During training, a CNN-based encoder maps UV texture maps into the appearance code ($\alpha$). This helps the network "memorize" the vast amount of photometric detail required for large-scale datasets.
  • RefineNet: To push the realism further, the coarse output from the NeRF is passed through a Pix2PixHD-based RefineNet to add high-frequency details and textures.
  • Landmark-based Sampling: Instead of sampling rays uniformly, the training process focuses more heavily on key facial features (eyes, nose, mouth) to ensure these critical areas are reconstructed with high fidelity.

Datasets

The model was trained and evaluated on two large-scale 3D face datasets:

  1. FaceScape: Contains 359 subjects, each captured in 20 different expressions.
  2. HeadSpace: Contains 1,004 subjects in a neutral expression.

These datasets provided the multi-view imagery and 3D priors (like PCA shape factors) necessary to train a model capable of representing a wide variety of human identities.

Key Insights and Ideas

  • Decoupling is Key: The authors discovered that merging shape and appearance into a single "identity" code made it too difficult for the MLP to learn. Separating them was essential for scaling to hundreds of subjects.
  • PCA vs. Learnable Codes: The paper finds that PCA-based codes are superior for representing shape because they capture geometric similarities between people. Conversely, expression is better handled by learnable codes because the categories (e.g., "smile," "frown") are discrete and easier for the network to parse without a linear prior.
  • Warping vs. Concatenation: While many dynamic NeRFs use a "deformation field" (warping) to handle motion, MoFaNeRF finds that simple concatenation of expression codes works better for large-scale, multi-subject datasets where learning a single continuous warping field is extremely difficult.

Key Results

MoFaNeRF significantly outperforms both traditional 3DMMs and recent implicit models like i3DMM in terms of image quality.

  • Quantitative: MoFaNeRF achieved a PSNR of 31.49 dB, compared to 27.96 dB for the standard FaceScape 3DMM. It also showed superior performance in structural similarity (SSIM) and perceptual metrics (LPIPS).
  • Qualitative: The model excels at "side-view" synthesis. While GAN-based methods often produce artifacts when rotated to large angles, MoFaNeRF maintains identity and shape even at $\pm 90^\circ$ rotations.
  • Applications: The model supports smooth morphing between different identities and expressions by simply interpolating the input codes in the vector space.
Fig. 1. We propose MoFaNeRF, which is a parametric model that can synthesize free- view images by fitting to a single image or generating from a random code. The syn- thesized face is morphable that can be rigged to a certain expression and be edited to a certain shape and appearance.

Limitations and Future Work

Despite its strengths, MoFaNeRF has a few hurdles to clear:

  • No Explicit Geometry: Because it is a radiance field, it does not explicitly generate a 3D mesh, making it harder to drop into traditional game engines or animation pipelines.
  • Lighting Sensitivity: The current fitting process works best under diffused lighting. The authors suggest that integrating an explicit illumination model is a vital next step.
  • Generalization: While it handles a wide range of faces, its performance can degrade when presented with identities (e.g., specific skin tones or extreme facial shapes) that fall far outside the training distribution of FaceScape and HeadSpace.

Summary updated

Abstract

We propose a parametric model that maps free-view images into a vector space of coded facial shape, expression and appearance with a neural radiance field, namely Morphable Facial NeRF. Specifically, MoFaNeRF takes the coded facial shape, expression and appearance along with space coordinate and view direction as input to an MLP, and outputs the radiance of the space point for photo-realistic image synthesis. Compared with conventional 3D morphable models (3DMM), MoFaNeRF shows superiority in directly synthesizing photo-realistic facial details even for eyes, mouths, and beards. Also, continuous face morphing can be easily achieved by interpolating the input shape, expression and appearance codes. By introducing identity-specific modulation and texture encoder, our model synthesizes accurate photometric details and shows strong representation ability. Our model shows strong ability on multiple applications including image-based fitting, random generation, face rigging, face editing, and novel view synthesis. Experiments show that our method achieves higher representation ability than previous parametric models, and achieves competitive performance in several applications. To the best of our knowledge, our work is the first facial parametric model built upon a neural radiance field that can be used in fitting, generation and manipulation. The code and data is available at https://github.com/zhuhao-nju/mofanerf.
BibTeX
@inproceedings{zhuang2022mofanerf,
  author = {Yiyu Zhuang and Hao Zhu and Xusen Sun and Xun Cao},
  title = {{MoFaNeRF: Morphable Facial Neural Radiance Field}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=706FLvdkk84},
}
Figure