Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space

Haoyu Wang, Haozhe Wu, Junliang Xing, Jia Jia

Abstract

Creating realistic 3D facial animation is crucial for various applications in the movie production and gaming industry, especially with the burgeoning demand in the metaverse. However, prevalent methods such as blendshape-based approaches and facial rigging techniques are time-consuming, labor-intensive, and lack standardized configurations, making facial animation production challenging and costly. In this paper, we propose a novel self-supervised framework, Versatile Face Animator, which combines facial motion capture with motion retargeting in an end-to-end manner, eliminating the need for blendshapes or rigs. Our method has the following two main characteristics: 1) we propose an RGBD animation module to learn facial motion from raw RGBD videos by hierarchical motion dictionaries and animate RGBD images rendered from 3D facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D characters regardless of their topology, textures, blendshapes, and rigs; and 2) we introduce a mesh retarget module to utilize RGBD animation to create 3D facial animation by manipulating facial mesh with controller transformations, which are estimated from dense optical flow fields and blended together with geodesic-distance-based weights. Comprehensive experiments demonstrate the effectiveness of our proposed framework in generating impressive 3D facial animation results, highlighting its potential as a promising solution for the cost-effective and efficient production of facial animation in the metaverse.

Introduction

In the world of movie production and gaming, creating realistic 3D facial animation is a significant bottleneck. The industry standard relies on blendshapes (predefined facial expressions) or rigging (digital "bones" and controllers). These methods are labor-intensive—often requiring hundreds of manual expressions for a single character—and lack interoperability; a rig designed for a human character typically cannot be applied to a stylized or non-human character without extensive manual re-mapping.

Versatile Face Animator (VFA) addresses these challenges with a self-supervised framework that can animate any 3D facial mesh using only raw RGBD (color + depth) video. By eliminating the need for manual rigging, VFA enables cost-effective, high-fidelity facial performance transfer across characters with completely different topologies.

Methodology

The VFA framework consists of two primary components that operate in an end-to-end pipeline: the RGBD Animation Module and the Mesh Retarget Module.

Overview of the Versatile Face Animator framework, illustrating the end-to-end pipeline from RGBD input to 3D animation output.

RGBD Animation Module

This module decomposes facial motion into a "Motion Dictionary"—a set of basic transformations. A key innovation is the introduction of a Depth Motion Dictionary. Unlike previous models that treat depth as a standard color channel, VFA specifically models how pixels move along the Z-axis. This allows the system to capture nuances such as the protrusion of lips or the movement of the nose relative to the camera. The module uses a distilled flow generator to estimate dense optical flow between the source image and the animated frame without requiring manual labels.

Mesh Retarget Module

This module translates 2D motion into 3D mesh deformations. It automatically identifies landmarks on the mesh to act as "controllers." To ensure natural deformation, the framework employs geodesic weighting. By using geodesic distance (distance along the surface of the skin) rather than Euclidean distance, the system prevents artifacts like "wave-lip," where moving the lower lip accidentally pulls the upper lip along with it.

Experiments and Results

The model was pre-trained on the VoxCeleb and VoxCeleb2 datasets and fine-tuned on MMFace4D, a large-scale facial RGBD video dataset. VFA was evaluated against state-of-the-art 2D animation methods (FOMM, OSFV, and DaGAN) adapted for RGBD.

Quantitative Performance

VFA achieved superior results across several metrics:

  • Reconstruction Quality: The model attained the lowest L1 error and LPIPS (perceptual loss) scores, particularly excelling in depth accuracy.
  • Semantic Consistency: Using the Average Keypoint Distance (AKD) metric, VFA proved more accurate at transferring the driver's actual expression to the target.

Qualitative Versatility

Qualitative results demonstrate the model's ability to successfully animate diverse characters—including women, children, and non-human aliens—while maintaining the identity of the target mesh and the semantic integrity of the driver's performance.

Cross-identity animation results demonstrating the framework's versatility across different character topologies, including stylized and non-human avatars.

Conclusion and Future Work

Versatile Face Animator provides a promising solution for the efficient production of 3D facial animation. By leveraging RGBD data and geodesic-based retargeting, it bypasses the traditional requirements for rigs and blendshapes.

Future work will focus on addressing current limitations, such as handling heavy occlusions and modeling fine details like pupil movement and tongue dynamics. The authors also plan to further improve the visual quality of generated frames to handle even more extreme stylized characters.

Summary updated

Abstract

Creating realistic 3D facial animation is crucial for various applications in the movie production and gaming industry, especially with the burgeoning demand in the metaverse. However, prevalent methods such as blendshape-based approaches and facial rigging techniques are time-consuming, labor-intensive, and lack standardized configurations, making facial animation production challenging and costly. In this paper, we propose a novel self-supervised framework, Versatile Face Animator, which combines facial motion capture with motion retargeting in an end-to-end manner, eliminating the need for blendshapes or rigs. Our method has the following two main characteristics: 1) we propose an RGBD animation module to learn facial motion from raw RGBD videos by hierarchical motion dictionaries and animate RGBD images rendered from 3D facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D characters regardless of their topology, textures, blendshapes, and rigs; and 2) we introduce a mesh retarget module to utilize RGBD animation to create 3D facial animation by manipulating facial mesh with controller transformations, which are estimated from dense optical flow fields and blended together with geodesic-distance-based weights. Comprehensive experiments demonstrate the effectiveness of our proposed framework in generating impressive 3D facial animation results, highlighting its potential as a promising solution for the cost-effective and efficient production of facial animation in the metaverse.
BibTeX
@inproceedings{wang2023versatile,
  author = {Haoyu Wang and Haozhe Wu and Junliang Xing and Jia Jia},
  title = {{Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612065},
}
Figure