TL;DR
This paper introduces a novel approach to creating realistic, real-time 3D avatars that can mimic facial expressions and movements. By leveraging a set of captured images and a unique image-based representation, the method effectively handles complex hairstyles and headwear, offering a more accessible and expressive alternative to traditional geometry-based avatars.
What Problem Does This Paper Address?
Creating personalized 3D avatars that can realistically animate with fine-scale facial details, including dynamic hairstyles, has been a significant challenge. Existing methods often struggle with hair and headwear, either simplifying them to textures or requiring complex, expensive capture setups. The goal is to enable users to create their own dynamic avatars from readily available hardware like webcams, capturing their unique appearance and enabling expressive animations driven by an actor's performance.
Methods
The core of this work lies in an "image-based representation" for dynamic 3D avatars. Instead of relying solely on geometric models, the avatar's appearance is primarily derived from a set of captured images of the user.
Here's a breakdown of their approach:
- Image Acquisition: A user provides a set of sparsely captured images (around 32) covering various head poses and facial expressions. This is done using a standard webcam.
- Avatar Construction:
- Face Blendshape Model: A coarse geometric model of the face is created by fitting a generic blendshape model to the captured images. This model captures the basic shape and expression dynamics.
- Morphable Hair Model: This is a key innovation. Instead of modeling individual hair strands, they create a "morphable hair model" that approximates the dynamic hair geometry. This is achieved through a multi-stage process involving single-image depth estimation, joint optimization to enforce inter-image consistency, and finally, constructing a topologically consistent morphable model. This approach allows for efficient handling of various hairstyles.
- Other Components: Eyes, teeth, and the body are represented using simpler "billboards" (2D images that always face the camera).
- Real-time Animation:
- Face Tracking: A real-time face tracker captures the facial motion of an arbitrary actor.
- Geometry Generation: The captured facial motion parameters are used to construct the coarse 3D geometry of the avatar (face and hair) for the current frame.
- Image Warping and Blending: The original captured images of the user are then warped and blended together, guided by the generated coarse geometry. This process ensures that the avatar's appearance matches the current pose and expression while retaining fine-scale details like wrinkles and folds, which are implicitly present in the captured images. A novel weighting strategy is used to ensure smooth blending of different image contributions.
Datasets
The paper does not explicitly mention using publicly available datasets for training or evaluation. Instead, they collect their own dataset by capturing images of users with a webcam. The "FaceWarehouse database" is mentioned as a source for initial 3D face models during the avatar construction phase.
Key Insights and Ideas
- Image-Based Representation as a Unified Solution: The core insight is that using captured images as the primary representation for an avatar, guided by coarse geometry, can naturally handle fine-scale details (wrinkles, pores) and complex elements like hair and headwear without explicit, separate modeling for each.
- Morphable Hair Model for Efficiency: The development of a "morphable hair model" that approximates dynamic hair geometry, rather than modeling individual strands, is crucial for achieving real-time performance and handling diverse hairstyles. This avoids the computational cost of strand-based hair simulation.
- Inter-Image Depth Optimization for Hair: The joint optimization of depth maps across multiple images is key to obtaining a consistent and accurate coarse hair geometry, overcoming the limitations of single-image depth estimation.
- Seamless Integration of Components: The method effectively integrates the image-based face and hair with simpler billboard representations for eyes and teeth, creating a cohesive avatar.
Key Results
- Real-time Performance: The system achieves real-time animation at 25 frames per second on a standard workstation, driven by a CPU-based face tracker.
- High-Fidelity Details: The avatars exhibit fine-scale facial details such as folds and wrinkles, which are implicitly captured from the input images.
- Effective Hair and Headwear Handling: The approach demonstrates successful animation of avatars with various hairstyles and headwear, a significant improvement over previous methods.
- Realistic Animation: Qualitative results show that the animated avatars closely match the captured video frames of the user, conveying realistic appearance and motion.
- Memory Efficiency: A completed image-based avatar requires approximately 50MB of storage.
Limitations and Future Work
- Limited Rotation: The authors acknowledge that their current pipeline does not support full 360° rotations around the neck. They suggest that incorporating more images, especially from the back view, could address this.
- Hair Deformation Complexity: While the morphable hair model is effective, it may not fully capture highly complex hair deformations, especially those involving significant occlusion changes. Strand-based hair representations might be necessary for such scenarios.
- Relighting: The current system warps and blends captured images without explicitly considering relighting effects. Future work could involve intrinsic decomposition of images to factor out lighting and enable subsequent relighting.
- 360° Rotations: The avatars do not support full 360° rotations around the neck, suggesting that incorporating more data or a different approach for the back of the head might be necessary.
Summary updated