TL;DR
MoFaNeRF is the first facial parametric model built entirely upon a Neural Radiance Field (NeRF) that supports image fitting, random generation, and interactive manipulation. By disentangling facial shape, expression, and appearance into a unified vector space, it achieves photo-realistic synthesis of complex details like pupils and beards that traditional mesh-based models struggle to capture.
What Problem Does This Paper Address?
For decades, the 3D Morphable Model (3DMM) has been the gold standard for modeling human faces. It allows researchers to represent faces using a compact set of parameters for shape and texture, making it easy to "rig" a face into new expressions or "fit" it to a single photo. However, 3DMMs rely on mesh-based rendering, which often produces "uncanny" results. They struggle with non-Lambertian surfaces (like the wetness of eyes) and fine-grained geometry (like individual beard hairs or the inside of a mouth).
On the other hand, Neural Radiance Fields (NeRF) have revolutionized photo-realistic view synthesis. While NeRFs produce stunningly realistic images, they are typically "baked" into a specific scene or person. Making a NeRF "morphable"—allowing it to represent thousands of different people and expressions in a single, editable model—is a significant technical challenge. MoFaNeRF bridges this gap, combining the editability of 3DMMs with the rendering power of NeRF.
Methods
The core of MoFaNeRF is a parametric mapping function that takes five inputs: a 3D position ($x$), a viewing direction ($d$), and three disentangled codes for shape ($\beta$), appearance ($\alpha$), and expression ($\epsilon$).
Key technical innovations include:
- Identity-Specific Modulation (ISM): The authors found that standard MLPs tend to "average out" expressions across different people. ISM uses the shape code to modulate the expression code, capturing the unique way an individual's face moves.
- Texture Encoding Module (TEM): During training, a CNN-based encoder maps UV texture maps into the appearance code ($\alpha$). This helps the network "memorize" the vast amount of photometric detail required for large-scale datasets.
- RefineNet: To push the realism further, the coarse output from the NeRF is passed through a Pix2PixHD-based RefineNet to add high-frequency details and textures.
- Landmark-based Sampling: Instead of sampling rays uniformly, the training process focuses more heavily on key facial features (eyes, nose, mouth) to ensure these critical areas are reconstructed with high fidelity.
Datasets
The model was trained and evaluated on two large-scale 3D face datasets:
- FaceScape: Contains 359 subjects, each captured in 20 different expressions.
- HeadSpace: Contains 1,004 subjects in a neutral expression.
These datasets provided the multi-view imagery and 3D priors (like PCA shape factors) necessary to train a model capable of representing a wide variety of human identities.
Key Insights and Ideas
- Decoupling is Key: The authors discovered that merging shape and appearance into a single "identity" code made it too difficult for the MLP to learn. Separating them was essential for scaling to hundreds of subjects.
- PCA vs. Learnable Codes: The paper finds that PCA-based codes are superior for representing shape because they capture geometric similarities between people. Conversely, expression is better handled by learnable codes because the categories (e.g., "smile," "frown") are discrete and easier for the network to parse without a linear prior.
- Warping vs. Concatenation: While many dynamic NeRFs use a "deformation field" (warping) to handle motion, MoFaNeRF finds that simple concatenation of expression codes works better for large-scale, multi-subject datasets where learning a single continuous warping field is extremely difficult.
Key Results
MoFaNeRF significantly outperforms both traditional 3DMMs and recent implicit models like i3DMM in terms of image quality.
- Quantitative: MoFaNeRF achieved a PSNR of 31.49 dB, compared to 27.96 dB for the standard FaceScape 3DMM. It also showed superior performance in structural similarity (SSIM) and perceptual metrics (LPIPS).
- Qualitative: The model excels at "side-view" synthesis. While GAN-based methods often produce artifacts when rotated to large angles, MoFaNeRF maintains identity and shape even at $\pm 90^\circ$ rotations.
- Applications: The model supports smooth morphing between different identities and expressions by simply interpolating the input codes in the vector space.
Limitations and Future Work
Despite its strengths, MoFaNeRF has a few hurdles to clear:
- No Explicit Geometry: Because it is a radiance field, it does not explicitly generate a 3D mesh, making it harder to drop into traditional game engines or animation pipelines.
- Lighting Sensitivity: The current fitting process works best under diffused lighting. The authors suggest that integrating an explicit illumination model is a vital next step.
- Generalization: While it handles a wide range of faces, its performance can degrade when presented with identities (e.g., specific skin tones or extreme facial shapes) that fall far outside the training distribution of FaceScape and HeadSpace.
Summary updated