TL;DR
This paper introduces MA-NeRF, a novel framework for synthesizing high-fidelity, drivable 3D face avatars from sparse image inputs. By integrating 3D Morphable Models (3DMM) with Neural Radiance Fields (NeRF), MA-NeRF effectively handles unseen facial expressions and improves generalization, overcoming limitations of existing methods.
What Problem Does This Paper Address?
Creating realistic 3D face avatars from limited visual data is a challenging task. Traditional parametric models like 3DMMs struggle to capture fine-grained details present in input images. While recent NeRF-based approaches excel at novel view synthesis, they often fail to generalize to facial expressions not seen during training, leading to artifacts. This paper aims to bridge this gap by developing a method that can reconstruct detailed, controllable 3D face avatars from sparse images and robustly handle a wide range of expressions.
Methods
MA-NeRF builds upon the Neural Radiance Fields (NeRF) framework but introduces several key innovations to address the aforementioned problems.
- 3DMM Integration: The core idea is to leverage the structured prior information from 3DMMs. Instead of relying solely on NeRF's implicit representation, MA-NeRF incorporates 3DMM parameters (expression, identity, pose) to guide the deformation of the neural radiance field.
- Structured Displacement Feature: To handle unseen expressions, the authors introduce a "structured displacement feature." This feature uses the vertex displacement information from 3DMMs, anchored and diffused using a SparseConvNet. This creates a continuous displacement volume that provides a motion prior, helping the model generalize better to new expressions.
- Semantic-Aware Learning: Recognizing that different facial components (eyes, mouth, nose) have distinct appearances and motion patterns, MA-NeRF employs a "semantic-aware learning module." This module projects 3DMM vertices onto a semantic graph and anchors learnable latent codes to these vertices, allowing for more nuanced control and better representation of facial semantics.
- Canonical Facial Model: The framework maintains a canonical facial model, where all observed points are transformed into a consistent space. This is achieved by decoupling rigid head pose transformations from non-rigid expression deformations.
Datasets
The paper evaluates MA-NeRF on publicly released video datasets, specifically mentioning AD-NeRF [16] and NerFACE [4]. These datasets consist of videos captured from a single stationary camera. The authors also mention using a sparse set of images (around 30-80 frames) for training and 25 images for testing per subject.
Key Insights and Ideas
- Motion Prior is Crucial for Expression Generalization: The paper strongly suggests that incorporating explicit motion priors, derived from 3DMMs, is essential for NeRF-based models to generalize to unseen facial expressions.
- Structured Features Enhance Robustness: By creating structured displacement and semantic features, the model gains a deeper understanding of facial geometry and semantics, leading to more robust synthesis.
- 3DMMs as a Bridge: 3DMMs, despite their limitations in capturing fine details, serve as a valuable structured prior that can significantly enhance the performance and generalization of implicit neural representations like NeRF.
- Decoupling Rigid and Non-Rigid Motion: Separating head pose (rigid) from facial expressions (non-rigid) is an effective strategy for building dynamic avatar models.
Key Results
- Quantitative Performance: MA-NeRF achieves state-of-the-art results on various metrics, including PSNR, SSIM, and LPIPS, outperforming methods like Neural Body and NerFACE, especially in sparse input settings. For instance, their reported PSNR is 24.08, SSIM is 0.923, and LPIPS is 0.042.
- Qualitative Improvements: The qualitative comparisons demonstrate that MA-NeRF generates images with higher quality, fewer artifacts, and better handling of unseen expressions compared to existing methods. The paper shows examples where MA-NeRF successfully synthesizes expressions under new poses, while other methods struggle.
- Ablation Study Validation: Ablation studies confirm the effectiveness of each proposed component: removing semantic features or expression parameters leads to performance degradation, and the inclusion of hard surface and canonical edge losses improves rendering quality by reducing fog and halo effects.
Limitations and Future Work
- Rendering Speed: The authors acknowledge that rendering a single frame takes several seconds, which is a limitation for real-time applications.
- Editable Operations: The current model does not support further editable operations on the synthesized face avatars.
- Future Work: The paper implicitly suggests that improving rendering speed and enabling more interactive editing capabilities would be valuable future research directions.
Summary updated