MoFaNeRF: Morphable Facial Neural Radiance Field

Yiyu Zhuang, Hao Zhu, Xusen Sun, Xun Cao

MoFaNeRF: Morphable Facial Neural Radiance Field

Introduction

For decades, the 3D Morphable Model (3DMM) has served as the standard for parametric facial modeling. By mapping facial shape and texture into a low-dimensional vector space, 3DMMs enable tasks such as expression rigging and 3D model fitting from 2D images. However, these mesh-based approaches struggle to represent non-Lambertian surfaces, such as the specular reflections in eyes, or complex geometries like hair, eyelashes, and beards.

Conversely, Neural Radiance Fields (NeRF) have achieved unprecedented photo-realistic rendering. Yet, NeRFs are typically scene-specific, requiring separate training for each individual. MoFaNeRF addresses this by introducing the first facial parametric model built entirely on NeRF, enabling a unified space for fitting, generating, and manipulating thousands of identities and expressions.

The MoFaNeRF Framework

MoFaNeRF defines a parametric model as a function mapping a 3D position ($x$), viewing direction ($d$), and three latent codes—shape ($\beta$), appearance ($\alpha$), and expression ($\epsilon$)—to color and density.

Overview of MoFaNeRF capabilities, including fitting, free-view synthesis, rigging, and editing.

Several key technical innovations facilitate this representation:

  • Texture Encoding Module (TEM): A CNN-based encoder compresses UV texture maps from 3D scans into the appearance code ($\alpha$), assisting the MLP in learning high-frequency details like pores and wrinkles.
  • Identity-Specific Modulation (ISM): To ensure expressions are anatomically correct across different face shapes, an ISM module modulates the expression code based on the shape code.
  • RefineNet: A post-processing 2D CNN sharpens the final output to overcome the inherent blurriness of some NeRF results.
  • Landmark-based Sampling: Training focuses on critical facial regions (eyes, nose, mouth) using 2D landmarks to improve local fidelity.

Key Insights and Results

The shift from mesh-based to volumetric representations allows MoFaNeRF to capture details that traditional models miss, such as the "wetness" of eyes and the complex interior of the mouth. By disentangling shape, appearance, and expression, the model supports sophisticated face editing, such as transferring skin tone or expressions between different identities.

Quantitatively, MoFaNeRF achieves a PSNR of 31.49 dB, significantly outperforming traditional 3DMMs (~28 dB) and prior implicit-function models. It maintains high fidelity during novel view synthesis, even at extreme 90-degree angles, where GAN-based competitors often fail. Furthermore, the continuous vector space allows for smooth interpolation between identities and expressions, enabling artifact-free morphing animations.

Limitations and Future Work

Despite its advantages, MoFaNeRF's volumetric nature makes it incompatible with traditional mesh-based game engines and VFX pipelines. The model is also sensitive to extreme lighting conditions and is bounded by the diversity of its training datasets (FaceScape and HeadSpace). Future research will focus on integrating explicit illumination models to allow for dynamic relighting and expanding the training data to encompass a broader range of ages and ethnicities.

Summary updated

BibTeX
@inproceedings{zhuang2022mofanerf,
  author = {Yiyu Zhuang and Hao Zhu and Xusen Sun and Xun Cao},
  title = {{MoFaNeRF: Morphable Facial Neural Radiance Field}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=706FLvdkk84},
}
Figure