Learning Speech-driven 3D Conversational Gestures from Video

Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, Christian Theobalt

Abstract

We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. Synthesis of conversational body gestures is a multi-modal problem since many similar gestures can plausibly accompany the same input speech. To synthesize plausible body gestures in this setting, we train a Generative Adversarial Network (GAN) based model that measures the plausibility of the generated sequences of 3D body motion when paired with the input audio features. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people. To this end, we apply state-of-the-art monocular approaches for 3D body and hand pose estimation as well as dense 3D face performance capture to the video corpus. In this way, we can train on orders of magnitude more data than previous algorithms that resort to complex in-studio motion capture solutions, and thereby train more expressive synthesis algorithms. Our experiments and user study show the state-of-the-art quality of our speech-synthesized full 3D character animations.

TL;DR

Researchers have developed the first system to jointly synthesize synchronized 3D body gestures, hand movements, and facial expressions of a virtual character directly from speech. By leveraging a massive new dataset of "in-the-wild" videos and a GAN-based architecture, the model produces animations that are significantly more natural and better synchronized than previous state-of-the-art methods.

What Problem Does This Paper Address?

Creating realistic animations for virtual characters is a notorious bottleneck in game development and VR. Traditionally, this requires either expensive motion capture (mocap) sessions or thousands of hours of manual work by expert animators.

Automating this via "speech-to-gesture" is difficult because the relationship is multi-modal: a single sentence could be accompanied by many different, equally plausible gestures. Furthermore, most previous research focused on isolated parts of the problem—generating just the face, just the 2D upper body, or just the hands. This paper tackles the challenge of generating a holistic, synchronized 3D performance (face, head, body, and hands) using only audio as input.

Methods

The authors propose a deep learning pipeline consisting of two main components:

Overview of the proposed pipeline, showing training data creation from in-the-wild video and the subsequent speech-to-animation generation process.
  1. A 1D Convolutional Encoder-Decoder: The model takes audio features (MFCCs) as input. A shared encoder learns a latent representation of the speech, which is then passed to three specialized "decoder heads" that output 3D facial expression coefficients, 3D body/head poses, and 3D hand keypoints.
  2. Adversarial Training (GAN): To solve the "regression to the mean" problem (where models produce boring, tiny movements to minimize error), they use a Generative Adversarial Network. A discriminator is trained to distinguish between real audio-gesture pairs and generated ones. This forces the generator to produce more expressive, rhythmic, and "plausible" motions that actually sync with the audio's prosody.
The GAN-based architecture featuring a shared encoder and specialized decoders for face, body, and hands, alongside a discriminator to ensure motion plausibility.

Datasets

A major contribution of this work is the creation of a large-scale training corpus.

  • The Data: They used over 33 hours of "in-the-wild" videos of talk show hosts (e.g., Conan O'Brien, Ellen DeGeneres).
  • The Innovation: Instead of using studio mocap, they applied state-of-the-art monocular 3D reconstruction algorithms (like XNect for bodies and dense face trackers) to extract 3D ground truth directly from 2D video.
  • Scale: This allowed them to train on orders of magnitude more data than previous studio-based approaches, capturing the unique "idiosyncrasies" and personal styles of different speakers.

Key Insights and Ideas

  • Correlation is Key: By using a shared encoder for the face and hands, the model leverages the inherent biological correlation between facial expressions and manual gestures.
  • Plausibility over Accuracy: Because there isn't one "correct" gesture for a sentence, the authors argue that a discriminator (GAN) is better than standard loss functions (like L1/L2). The goal isn't to match a specific recorded frame, but to be plausible given the audio.
  • The "In-the-Wild" Advantage: The paper proves that noisy, automatically-extracted 3D data from YouTube videos is actually better for training expressive models than small amounts of "clean" studio data, simply because of the sheer volume and variety of gestures available in the wild.

Key Results

The researchers conducted extensive user studies to evaluate their results against baselines like LSTMs and 2D-to-3D lifting models.

  • Naturalness & Synchrony: In a 5-point scale user study, the proposed model scored 4.05 for naturalness and 4.00 for synchrony, significantly outperforming the next best baseline (3.54 and 3.78, respectively).
  • Lip Sync: Quantitatively, the model achieved lip-motion prediction errors on par with specialized face-only models, despite being a unified whole-body system.
  • Qualitative Success: The model successfully captures "beat gestures"—the rhythmic hand chops and head nods humans use to emphasize speech—without being explicitly programmed to do so.

Limitations and Future Work

  • Subject Specificity: Currently, the models are trained per-person. A model trained on Conan O'Brien won't necessarily produce "Ellen-like" gestures. Creating a universal, multi-speaker model is a logical next step.
  • Lack of Semantics: The model reacts to the sound (prosody/rhythm) of the speech, not the meaning. It doesn't know to point at a "ball" if the speaker says the word "ball." Integrating text-based semantic understanding could improve the relevance of the gestures.
  • Lower Body: The current system focuses on the upper body (torso, arms, hands, head). For full-character animation, the authors currently fuse their results with pre-recorded leg animations.

Summary updated

Abstract

We propose the first approach to automatically and jointly synthesize both the synchronous 3D conversational body and hand gestures, as well as 3D face and head animations, of a virtual character from speech input. Our algorithm uses a CNN architecture that leverages the inherent correlation between facial expression and hand gestures. Synthesis of conversational body gestures is a multi-modal problem since many similar gestures can plausibly accompany the same input speech. To synthesize plausible body gestures in this setting, we train a Generative Adversarial Network (GAN) based model that measures the plausibility of the generated sequences of 3D body motion when paired with the input audio features. We also contribute a new way to create a large corpus of more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people. To this end, we apply state-of-the-art monocular approaches for 3D body and hand pose estimation as well as dense 3D face performance capture to the video corpus. In this way, we can train on orders of magnitude more data than previous algorithms that resort to complex in-studio motion capture solutions, and thereby train more expressive synthesis algorithms. Our experiments and user study show the state-of-the-art quality of our speech-synthesized full 3D character animations.
BibTeX
@inproceedings{habibie2021learning,
  author = {Ikhsanul Habibie and Weipeng Xu and Dushyant Mehta and Lingjie Liu and Hans-Peter Seidel and Gerard Pons-Moll and Mohamed Elgharib and Christian Theobalt},
  title = {{Learning Speech-driven 3D Conversational Gestures from Video}},
  booktitle = {IVA},
  year = {2021},
  doi = {10.1145/3472306.3478335},
}
Figure