Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space

Haoyu Wang, Haozhe Wu, Junliang Xing, Jia Jia

TL;DR

Versatile Face Animator (VFA) is a new self-supervised framework that can animate any 3D facial mesh using nothing but raw RGBD (color + depth) video. By eliminating the need for manual rigging or complex blendshapes, VFA allows for cost-effective, high-fidelity facial performance transfer across characters with completely different topologies.

What Problem Does This Paper Address?

In the world of movie production and gaming, creating realistic 3D facial animation is a massive bottleneck. The industry standard relies on blendshapes (predefined facial expressions) or rigging (digital "bones" and controllers). These methods are incredibly labor-intensive—sometimes requiring hundreds of manual expressions for a single character—and they don't play well together. A rig designed for a human character won't work for an orc or an alien without significant manual re-mapping.

The authors aim to bridge the gap between 2D face reenactment (which is easy but lacks 3D structure) and 3D facial retargeting (which is high-quality but expensive). They want a system that can take a video of a person and "drive" a 3D model automatically, regardless of how that model was built.

Methods

VFA consists of two primary components that work in an end-to-end pipeline:

  1. RGBD Animation Module:
    • This module learns to decompose facial motion into a "Motion Dictionary"—a set of basic transformations.
  2. The Innovation: Unlike previous models that treat depth as just another color channel, VFA introduces a Depth Motion Dictionary. This specifically models how pixels move along the Z-axis (depth), allowing the system to capture nuances like a nose moving closer to the camera or the subtle protrusion of lips.
  3. It uses a "distilled" flow generator to estimate dense optical flow between the source image and the animated frame without needing manual labels.
  4. Mesh Retarget Module:
    • This module takes the 2D motion and applies it to the 3D mesh. It automatically selects landmarks on the mesh to act as "controllers."
    • Geodesic Weighting: To ensure the mesh deforms naturally, the authors use geodesic distance (distance along the surface of the skin) rather than simple Euclidean distance (straight-line distance) to calculate how much a controller affects a specific vertex. This prevents "wave-lip" artifacts, where moving the lower lip accidentally pulls the upper lip along with it.

Datasets

  • VoxCeleb & VoxCeleb2: Used for initial pre-training. These are massive datasets of human speech videos.
  • MMFace4D: A large-scale facial RGBD video dataset captured by Azure Kinect V2. This was used for fine-tuning and testing the model's ability to handle depth information.

Key Insights and Ideas

  • Depth is not just a 3rd channel: Standard 2D animation models often fail in 3D because they don't understand that depth changes are a specific type of motion. By creating a dedicated dictionary for depth transformations, VFA achieves much higher reconstruction fidelity.
  • Self-Supervision via Distillation: The model learns to estimate optical flow by "watching" its own generator. This allows it to learn complex motion correspondence without any human-annotated "ground truth" flow maps.
  • Topology Independence: Because the system drives the mesh via dense flow and surface-based weights, it doesn't care if your 3D model has 5,000 vertices or 50,000, or if it's a human or a stylized monster.

Key Results

VFA was compared against state-of-the-art 2D animation methods (FOMM, OSFV, and DaGAN) adapted for RGBD:

  • Reconstruction Quality: VFA achieved the lowest L1 error and LPIPS (perceptual loss) scores on the MMFace4D and VoxCeleb datasets, particularly excelling in depth accuracy.
  • Semantic Consistency: Using the Average Keypoint Distance (AKD) metric, VFA proved more accurate at transferring the actual expression of the driver to the target.
  • Cross-Identity Transfer: In "user studies," participants consistently rated VFA's animations as more natural and semantically consistent than the baselines.
  • Visual Fidelity: Qualitative results show the model successfully animating diverse characters—including a woman, a child, and a non-human alien—while maintaining the identity of the target mesh.

Limitations and Future Work

  • Occlusions: Like most flow-based methods, the model struggles when parts of the face are heavily occluded (e.g., a hand moving in front of the face).
  • Unseen Avatars: While versatile, extremely unusual or "out-of-distribution" facial structures may still require more diverse training data to animate perfectly.
  • Fine Details: The current mesh retargeting doesn't separately model the eyes (pupil movement) or the tongue. If the target mesh doesn't have these parts modeled as separate entities, the system cannot "stick out the tongue" or "roll the eyes."
  • Future Direction: The authors plan to improve the quality of the generated RGBD frames and increase the versatility of the framework to handle even more extreme stylized characters.

Summary updated

Abstract

Creating realistic 3D facial animation is crucial for various applications in the movie production and gaming industry, especially with the burgeoning demand in the metaverse. However, prevalent methods such as blendshape-based approaches and facial rigging techniques are time-consuming, labor-intensive, and lack standardized configurations, making facial animation production challenging and costly. In this paper, we propose a novel self-supervised framework, Versatile Face Animator, which combines facial motion capture with motion retargeting in an end-to-end manner, eliminating the need for blendshapes or rigs. Our method has the following two main characteristics: 1) we propose an RGBD animation module to learn facial motion from raw RGBD videos by hierarchical motion dictionaries and animate RGBD images rendered from 3D facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D characters regardless of their topology, textures, blendshapes, and rigs; and 2) we introduce a mesh retarget module to utilize RGBD animation to create 3D facial animation by manipulating facial mesh with controller transformations, which are estimated from dense optical flow fields and blended together with geodesic-distance-based weights. Comprehensive experiments demonstrate the effectiveness of our proposed framework in generating impressive 3D facial animation results, highlighting its potential as a promising solution for the cost-effective and efficient production of facial animation in the metaverse.
BibTeX
@inproceedings{wang2023versatile,
  author = {Haoyu Wang and Haozhe Wu and Junliang Xing and Jia Jia},
  title = {{Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space}},
  booktitle = {ACM Multimedia},
  year = {2023},
  doi = {10.1145/3581783.3612065},
}

Loading related content...

Figure