TL;DR
This paper introduces a novel video-driven approach to generate high-quality, physically-based 3D facial assets for production. By combining a specialized capture system with a multi-VAE neural network, they achieve disentangled facial representations that enable realistic animation, cross-identity retargeting, and even stylized editing, bridging the gap between neural methods and production needs.
What Problem Does This Paper Address?
Creating realistic 3D dynamic human faces for film and games is a complex, labor-intensive process. Traditional pipelines rely on specialized hardware and manual artistry for geometry, texture, and motion. While recent neural methods automate parts of this process, they often lack explicit controls for artists and struggle to achieve production-level fidelity. This paper aims to bridge this gap by developing a learning-based system that can generate high-quality, animatable facial assets directly from video, offering artists more control and reducing manual effort.
Methods
The core of this work is a multi-Variational Autoencoder (VAE) framework designed to disentangle various facial attributes.
- Data Acquisition: They introduce a "Fast Capture Stage" (FaStage) that uses synchronized multi-view cameras and photometric lighting to capture dynamic facial geometry and physically-based textures (albedo, specular, normal maps) at high frame rates.
- Texture Stabilization: A novel unsupervised scheme using a U-Net predicts warp fields and wrinkle maps to stabilize temporal textures, mitigating jitter artifacts from the capture process.
- Disentangled Representation: Three separate VAEs are trained to learn latent representations for:
- Expression: A key innovation is a triplet supervision scheme that forces the VAE to disentangle expression from viewpoint, ensuring consistent expression representation across different camera angles.
- Geometry: A similar VAE disentangles facial geometry from head pose, allowing for robust geometry prediction even with varying head movements.
- Textures: A VAE models diffuse albedo, specular intensity, and normal maps, but importantly, it predicts "wrinkle maps" rather than direct textures. These maps represent deviations from a neutral texture and are crucial for achieving high-resolution (4K) dynamic textures and enabling editing.
- Latent Mapping: A global Multi-Layer Perceptron (MLP) maps the disentangled expression latent code to the geometry and texture latent codes, enabling the generation of consistent geometry and textures from an expression input.
- Inference: The system takes an input image, extracts the expression latent code, uses the MLP to predict geometry and texture latent codes, and then decodes these into physically-based facial assets.
Datasets
- Custom Dataset: The authors collected a new dataset using their FaStage system. This dataset consists of approximately 5 minutes of facial performance for each of three performers, captured at 24 fps, resulting in high-resolution (4K) dynamic facial geometry and physically-based textures.
- In-the-wild videos: For cross-identity retargeting, they utilize in-the-wild video footage.
Key Insights and Ideas
- Disentanglement is Key for Control: By explicitly disentangling expression from viewpoint and pose, the system gains robustness and allows for more predictable control over facial animation. This is a significant step beyond implicit neural representations.
- Wrinkle Maps for High-Res Dynamic Textures: Predicting wrinkle maps instead of direct textures is a clever way to preserve fine details and enable high-resolution (4K) dynamic texture generation, which is crucial for production quality.
- MLP as a Unified Expression Translator: Using a single MLP to map expression to both geometry and texture latent spaces ensures that these attributes remain consistent with each other, leading to more coherent facial assets.
- Bridging the Gap to Production: The combination of high-fidelity capture, disentangled representations, and the ability to generate physically-based assets makes this approach more amenable to existing production pipelines than previous neural methods.
Key Results
- High-Fidelity Capture: The FaStage system captures detailed facial geometry and 4K physically-based textures.
- Accurate Reconstruction: Quantitative comparisons show their method outperforms state-of-the-art methods in geometric reconstruction accuracy (e.g., RMSE of 1.516 mm vs. 2.237 mm for Laine et al. [2017]).
- Robustness: The disentanglement strategies lead to more stable and consistent results, especially when dealing with varying head poses and viewpoints.
- Cross-Identity Retargeting: The system successfully retargets facial motion from one individual to another, producing photo-realistic results.
- Stylized Editing: The disentangled nature of the assets allows for physically-based editing, enabling the creation of stylized avatars with novel appearances and expressions.
Limitations and Future Work
- Semantic Animation Controls: While the system offers more control than purely implicit methods, it still lacks explicit semantic controls for artists. The authors suggest alternatives like using rendered images from parametric models or mapping extracted explicit parameters.
- Performer-Specific Paradigm: The current approach is largely performer-specific, and handling extreme identity and environmental variations remains a challenge. Future work could focus on more generalizable expression networks.
- Missing Facial Components: The model primarily focuses on soft tissues and does not simulate teeth, gums, eyeballs, or eyelashes, which are necessary for full realism. Future work will incorporate eye movement and jaw adaptation.
- Hair Rendering: Realistic hair rendering is still a significant challenge in computer graphics, and the current model does not address it.
- Pore-Level Deformation: While wrinkle maps preserve details, they don't fully simulate phenomena like pore squeezing. Future work aims to incorporate warp-field-based texture inference for more nuanced pore-level deformation.
Summary updated