Abstract
Generating natural non-verbal behaviors for virtual agents remains a significant challenge in human-computer interaction. This paper introduces a deep learning framework that generates 3D human gestures from speech audio by first learning a compressed representation of human motion. Our proposed model, Aud2Repr2Pose, utilizes a Denoising Autoencoder to capture motion dynamics, which are then mapped from speech features using a recurrent neural network. While objective evaluations demonstrate that our method produces smoother and more realistic motion compared to direct-mapping baselines, subjective human studies reveal a surprising "equalizing" effect of simple post-processing. Temporal smoothing and hip-centering were found to level the perceived naturalness across models, highlighting critical considerations for the evaluation of synthetic gestures.
Introduction
As virtual agents and social robots are increasingly integrated into social environments, the demand for realistic non-verbal communication has grown. Traditional rule-based systems for gesture generation often lack the fluidity and nuance of human movement. Recent shifts toward data-driven deep learning models attempt to map speech audio directly to 3D joint coordinates. However, this high-dimensional mapping often results in jerky, unnatural motions that fail to synchronize with the temporal dynamics of human speech. This research investigates whether representation learning can bridge this gap and examines the impact of common post-processing techniques on human perception.
Methodology
The core contribution of this work is the Aud2Repr2Pose architecture. Unlike traditional models that attempt direct audio-to-pose mapping, our system employs a three-stage pipeline centered on representation learning.
- Motion Encoding: A Denoising Autoencoder (DAE) is trained to compress full-body 3D motion data, including joint velocities, into a lower-dimensional bottleneck representation ($Z$). This compression forces the network to prioritize essential motion dynamics over high-frequency noise.
- Speech-to-Representation: A recurrent neural network, utilizing Gated Recurrent Units (GRUs), is trained to map various speech audio features (such as MFCCs, spectrograms, or prosodic features) to the learned motion representation space.
- Decoding: During inference, the predicted representations are passed through the decoder to generate full 3D poses.
Experimental Setup
The models were trained and evaluated on two large-scale motion-capture datasets:
- Japanese Dataset: 171 minutes of motion capture (64 joints) and audio from interview settings.
- English Dataset: 202 minutes of motion capture (46 joints) and audio from impromptu speech by an Irish actor.
We experimented with multiple input modalities, including Mel-frequency cepstral coefficients (MFCCs), raw spectrograms, and prosodic features (pitch and energy), to determine their relative effectiveness in driving gesture generation.
Results
Objective Evaluation
The Aud2Repr2Pose model significantly outperformed baseline direct-mapping models across several objective metrics. It achieved a lower Average Position Error (7.66 cm vs. 8.07 cm) and a dramatic reduction in motion jerkiness (18.2 cm/s³ vs. 52.4 cm/s³). Furthermore, the speed profile of the generated gestures closely mirrored real human data, with a Hellinger distance of 0.133 compared to the baseline's 0.325.
Subjective Evaluation and the "Smoothing" Equalizer
Initial user studies using raw model outputs favored the proposed model, which was rated significantly more natural (4.79/7) than the baseline (4.16/7). However, a critical finding emerged when post-processing was applied. After applying temporal smoothing and hip-centering (to remove whole-body drift), human raters found no statistically significant difference in naturalness between the baseline, the proposed model, and even the ground-truth human motion. This reveals that simple post-processing can easily mask underlying model deficiencies.
Discussion and Conclusion
Our findings suggest that while representation learning is objectively superior for capturing motion dynamics, simple post-processing can effectively mask underlying model deficiencies in human-centric evaluations. This "equalizing" effect indicates that naturalness ratings may be sensitive to low-level signal properties like smoothness rather than high-level coordination.
A primary limitation of the current approach is its reliance on acoustic features alone, which limits the generation of semantically meaningful (iconic) gestures. Future work will focus on integrating text transcripts for semantic awareness and exploring probabilistic models to capture the inherent variability of human gesturing.
Summary updated