Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications

Karren D. Yang, Anurag Ranjan, Jen-Hao Rick Chang, Raviteja Vemulapalli, Oncel Tuzel

Researchers at Apple have addressed a fundamental limitation in computer animation: the "one-to-many" nature of human speech. While traditional models treat facial movement as a direct, deterministic byproduct of sound, real-world speech is influenced by unique personal styles, emotions, and linguistic nuances. This work introduces a probabilistic framework designed to capture the rich diversity of human expression while maintaining precise synchronization with driving audio.

The Challenge of Deterministic Animation

Traditional 3D facial animation models, such as VOCA or FaceFormer, are typically deterministic. They map a speech signal to a single, specific sequence of 3D face meshes. However, the relationship between speech and motion is inherently one-to-many; different speakers express the same words differently (inter-speaker variation), and even a single individual varies their expression based on mood or context (intra-speaker variation).

Deterministic models tend to produce the "mean" of all possible motions, which often results in robotic, stiff animations that lack the subtle upper-face dynamics necessary for realism. Furthermore, the field has historically been constrained by small datasets—often involving only 10-20 speakers—and a lack of metrics capable of measuring diversity rather than just accuracy to a single ground truth.

A Two-Stage Probabilistic Framework

To overcome these hurdles, the researchers proposed a hierarchical approach that separates motion discretization from temporal prediction.

Overview of the proposed framework, featuring Residual Vector Quantization (RVQ) for motion discretization and a two-stage auto-regressive model.

1. Residual Vector Quantization (RVQ)

Instead of predicting raw vertex positions, the model first learns to discretize 3D facial motion into a "codebook." By utilizing a coarse-to-fine hierarchy (RVQ), the system can represent complex, high-fidelity motions efficiently as a sequence of discrete codes.

2. Two-Stage Auto-Regressive (AR) Model

The synthesis process is split into two specialized components:

  • Temporal Model: This stage analyzes the input audio and the animation history to generate a context embedding.
  • Depth Model: Conditioned on the temporal context and a style reference clip, this stage predicts the specific RVQ codes for the current frame. Injecting style information at this second stage was found to be crucial for maintaining lip-sync precision while allowing for expressive variation.

3. Sampling and Distillation

To provide user control, the framework supports various sampling strategies, such as KNN-based sampling and SyncNet-based rejection sampling. These allow for a customizable trade-off between maximum diversity for creative use and maximum lip-sync precision for technical applications. For real-time deployment, the complex sampling process is distilled into a faster student network.

Scaling the Data Gap

A major contribution of this work is the creation of a massive new benchmark. While existing datasets like VOCASet and BIWI contain fewer than 1,200 sequences, the authors processed the VoxCeleb2 video corpus using state-of-the-art 3D face reconstruction (DECA and SPECTRE). This resulted in over 1 million sequences across 6,112 speakers, providing the scale required to learn true probabilistic distributions of human motion.

Key Results and Performance

The proposed model significantly outperformed existing baselines (VOCA, FaceFormer, MeshTalk) across several key metrics.

Quantitative evaluation comparing the proposed model against baselines across different mesh types and metrics, showing the trade-off between sync and diversity.
  • Realism and Diversity: The model achieved superior Frechet Distance (FD) scores, indicating that its generated motion distribution closely matches real human behavior.
  • Perceptual Preference: In user studies, participants overwhelmingly preferred the proposed model for both "Style Matching" and "Lip Realism."
  • Downstream Utility: One of the most striking findings was the utility of synthetic data. Training an Audio-Visual Speech Recognition (AVSR) model on these synthetic meshes yielded an 11.3% improvement in Word Error Rate (WER) compared to training on real video data, suggesting the synthetic meshes provide a "cleaner" learning signal than raw video.

Conclusion and Future Work

By moving from deterministic to probabilistic modeling, this research enables the synthesis of 3D facial animations that are both diverse and highly synchronized. While the model currently relies on monocular 3D reconstructions for its training data, future iterations could benefit from higher-precision multi-view captures. This technology holds significant promise for professional animation, gaming, and improving the robustness of speech-related AI tasks.

Summary updated

Abstract

We consider the task of animating 3D facial geometry from speech signal. Existing works are primarily deterministic, focusing on learning a one-to-one mapping from speech signal to 3D face meshes on small datasets with limited speakers. While these models can achieve high-quality lip articulation for speakers in the training set, they are unable to capture the full and diverse distribution of 3D facial motions that accompany speech in the real world. Importantly, the relationship between speech and facial motion is one-to-many, containing both inter-speaker and intra-speaker variations and necessitating a probabilistic approach. In this paper, we identify and address key challenges that have so far limited the development of probabilistic models: lack of datasets and metrics that are suitable for training and evaluating them, as well as the difficulty of designing a model that generates diverse results while remaining faithful to a strong conditioning signal as speech. We first propose large-scale benchmark datasets and metrics suitable for probabilistic modeling. Then, we demonstrate a probabilistic model that achieves both diversity and fidelity to speech, outperforming other methods across the proposed benchmarks. Finally, we showcase useful applications of probabilistic models trained on these large-scale datasets: we can generate diverse speech-driven 3D facial motion that matches unseen speaker styles extracted from reference clips; and our synthetic meshes can be used to improve the performance of downstream audio-visual models.
BibTeX
@inproceedings{yang2023probabilistic,
  author = {Karren D. Yang and Anurag Ranjan and Jen-Hao Rick Chang and Raviteja Vemulapalli and Oncel Tuzel},
  title = {{Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure