2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?

Teo Guichoux, Laure Soulier, Nicolas Obin, Catherine Pelachaud

Introduction

Generating realistic, speech-synchronized gestures for digital humans—often referred to as Embodied Conversational Agents (ECAs)—is a cornerstone of creating immersive human-computer interactions. However, the field faces a significant data bottleneck. While high-fidelity 3D Motion Capture (MoCap) provides the most accurate data, it is prohibitively expensive and difficult to scale. Consequently, researchers often turn to "in-the-wild" datasets harvested from video platforms like YouTube.

These datasets typically provide 2D skeletal data (X, Y coordinates). To achieve 3D outputs (X, Y, Z), researchers employ "lifting" models—temporal algorithms designed to estimate depth from 2D sequences. This creates a fundamental architectural question: is it more effective to train a model on 2D data and lift the final output to 3D, or to lift the dataset to 3D first and train the generative model directly on those estimated 3D coordinates?

Methodological Framework

To investigate this dimensionality dilemma, the researchers developed a comparative evaluation pipeline utilizing two state-of-the-art generative architectures:

  1. DiffGesture: A Denoising Diffusion Probabilistic Model (DDPM) that generates gestures by iteratively removing noise from a signal, known for its ability to capture complex distributions.
  2. Trimodal: A recurrent neural network (RNN) based model that maps speech audio and text transcripts directly into motion sequences.

For the lifting component, the study utilized VideoPose3D, a Temporal Convolutional Network (TCN). This model was used both to create the 3D training data from 2D sources and to lift the outputs of the 2D-trained generative models for final comparison. The experiment was designed to isolate the impact of dimensionality by comparing models trained on 2D coordinates against those trained on 3D coordinates, even testing if 3D training benefits models that are ultimately evaluated in a 2D space.

The "One-to-Many" Problem and Inductive Bias

A core theoretical contribution of this research is the identification of the "determinism trap." In gesture synthesis, a single 2D pose can theoretically correspond to multiple 3D configurations. However, most 3D lifting algorithms are deterministic; they are trained to provide a single, most-likely depth estimation for any given 2D input.

When a model is trained on 2D data and then lifted, it is subject to this deterministic bottleneck at the very end of the pipeline. Conversely, training directly on (even estimated) 3D data allows the generative model to learn the spatial relationships and depth dynamics as part of the motion synthesis itself. This avoids the "tightening" of the distribution that occurs when applying a deterministic lifter to a generated 2D sequence, resulting in gestures that are more diverse and less repetitive.

Experimental Results

The study utilized the TED Gesture-3D dataset, comprising 97 hours of talk show footage. While this dataset is technically "pseudo-ground-truth" (as the 3D poses were originally estimated via VideoPose3D), it serves as the current benchmark for in-the-wild gesture generation.

Objective Metrics

The 3D-trained models outperformed their 2D-lifted counterparts across all primary metrics:

  • Fréchet Gesture Distance (FGD): This metric measures the distance between the distribution of generated gestures and real human motion. DiffGesture trained in 3D achieved a score of 1.947, significantly better than the 3.121 achieved by the 2D-then-lifted version.
  • Beat Consistency (BC): This measures how well the motion aligns with the rhythmic peaks of the speech. 3D models demonstrated superior synchronization, scoring 0.678 compared to 0.551 for the 2D-lifted models.

Subjective User Study

A user study involving 60 participants corroborated the objective findings. Participants consistently rated the 3D-trained animations as more "human-like," "alive," and better synchronized with the audio. Notably, the preference for the 3D version of the Trimodal (RNN) model was particularly pronounced, suggesting that RNN architectures may struggle more with the artifacts introduced by late-stage 2D-to-3D conversion than diffusion-based models.

Discussion and Limitations

The findings suggest that learning the 3D structure of the human body—even from imperfect, estimated data—helps generative models understand the underlying mechanics of motion better than observing 2D silhouettes alone. Interestingly, models trained on 3D data performed better even when the results were projected back into 2D for evaluation, proving that depth information provides a vital regularizing effect on motion learning.

However, the authors note a "circularity" in current research: because the ground truth used for evaluation is itself the product of a 3D lifter, the evaluation may be biased toward the specific artifacts of that lifter. Future work is required to validate these findings against "true" 3D MoCap data and to explore more granular movements, such as finger articulations, which are currently lost in most 2D-to-3D lifting processes.

Conclusion

The research provides a clear recommendation for the field: train in 3D. Even when working with 2D video datasets, the intermediate step of lifting the data to 3D before training the generative model yields significantly higher quality, better synchronization, and more natural diversity than generating in 2D and lifting as a post-processing step.

Summary updated

Abstract

Co-speech gestures are fundamental for communication. The advent of recent deep learning techniques has facilitated the creation of lifelike, synchronous co-speech gestures for Embodied Conversational Agents. "In-the-wild" datasets, aggregating video content from platforms like YouTube via human pose detection technologies, provide a feasible solution by offering 2D skeletal sequences aligned with speech. Concurrent developments in lifting models enable the conversion of these 2D sequences into 3D gesture databases. However, it is important to note that the 3D poses estimated from the 2D extracted poses are, in essence, approximations of the ground-truth, which remains in the 2D domain. This distinction raises questions about the impact of gesture representation dimensionality on the quality of generated motions. Our study examines the effect of using either 2D or 3D joint coordinates as training data on the performance of speech-to-gesture deep generative models.
BibTeX
@inproceedings{guichoux2024d,
  author = {Teo Guichoux and Laure Soulier and Nicolas Obin and Catherine Pelachaud},
  title = {{2D or not 2D: How Does the Dimensionality of Gesture Representation Affect 3D Co-Speech Gesture Generation?}},
  booktitle = {IVA},
  year = {2024},
  doi = {10.1145/3652988.3673934},
}
Figure