Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation

Taras Kucherenko, Dai Hasegawa, Naoshi Kaneko, Gustav Eje Henter, Hedvig Kjellström

Abstract

Generating natural non-verbal behaviors for virtual agents remains a significant challenge in human-computer interaction. This paper introduces a deep learning framework that generates 3D human gestures from speech audio by first learning a compressed representation of human motion. Our proposed model, Aud2Repr2Pose, utilizes a Denoising Autoencoder to capture motion dynamics, which are then mapped from speech features using a recurrent neural network. While objective evaluations demonstrate that our method produces smoother and more realistic motion compared to direct-mapping baselines, subjective human studies reveal a surprising "equalizing" effect of simple post-processing. Temporal smoothing and hip-centering were found to level the perceived naturalness across models, highlighting critical considerations for the evaluation of synthetic gestures.

Introduction

As virtual agents and social robots are increasingly integrated into social environments, the demand for realistic non-verbal communication has grown. Traditional rule-based systems for gesture generation often lack the fluidity and nuance of human movement. Recent shifts toward data-driven deep learning models attempt to map speech audio directly to 3D joint coordinates. However, this high-dimensional mapping often results in jerky, unnatural motions that fail to synchronize with the temporal dynamics of human speech. This research investigates whether representation learning can bridge this gap and examines the impact of common post-processing techniques on human perception.

Comparison of different temporal modeling approaches for mapping audio sequences to skeletal motion.

Methodology

The core contribution of this work is the Aud2Repr2Pose architecture. Unlike traditional models that attempt direct audio-to-pose mapping, our system employs a three-stage pipeline centered on representation learning.

Overview of the proposed Aud2Repr2Pose framework, showing (a) motion representation learning, (b) speech-to-representation mapping, and (c) the combined inference model.
  1. Motion Encoding: A Denoising Autoencoder (DAE) is trained to compress full-body 3D motion data, including joint velocities, into a lower-dimensional bottleneck representation ($Z$). This compression forces the network to prioritize essential motion dynamics over high-frequency noise.
  2. Speech-to-Representation: A recurrent neural network, utilizing Gated Recurrent Units (GRUs), is trained to map various speech audio features (such as MFCCs, spectrograms, or prosodic features) to the learned motion representation space.
  3. Decoding: During inference, the predicted representations are passed through the decoder to generate full 3D poses.

Experimental Setup

The models were trained and evaluated on two large-scale motion-capture datasets:

  • Japanese Dataset: 171 minutes of motion capture (64 joints) and audio from interview settings.
  • English Dataset: 202 minutes of motion capture (46 joints) and audio from impromptu speech by an Irish actor.

We experimented with multiple input modalities, including Mel-frequency cepstral coefficients (MFCCs), raw spectrograms, and prosodic features (pitch and energy), to determine their relative effectiveness in driving gesture generation.

Results

Objective Evaluation

The Aud2Repr2Pose model significantly outperformed baseline direct-mapping models across several objective metrics. It achieved a lower Average Position Error (7.66 cm vs. 8.07 cm) and a dramatic reduction in motion jerkiness (18.2 cm/s³ vs. 52.4 cm/s³). Furthermore, the speed profile of the generated gestures closely mirrored real human data, with a Hellinger distance of 0.133 compared to the baseline's 0.325.

Subjective Evaluation and the "Smoothing" Equalizer

Initial user studies using raw model outputs favored the proposed model, which was rated significantly more natural (4.79/7) than the baseline (4.16/7). However, a critical finding emerged when post-processing was applied. After applying temporal smoothing and hip-centering (to remove whole-body drift), human raters found no statistically significant difference in naturalness between the baseline, the proposed model, and even the ground-truth human motion. This reveals that simple post-processing can easily mask underlying model deficiencies.

Discussion and Conclusion

Our findings suggest that while representation learning is objectively superior for capturing motion dynamics, simple post-processing can effectively mask underlying model deficiencies in human-centric evaluations. This "equalizing" effect indicates that naturalness ratings may be sensitive to low-level signal properties like smoothness rather than high-level coordination.

A primary limitation of the current approach is its reliance on acoustic features alone, which limits the generation of semantically meaningful (iconic) gestures. Future work will focus on integrating text transcripts for semantic awareness and exploring probabilistic models to capture the inherent variability of human gesturing.

Summary updated

Abstract

This paper presents a novel framework for speech-driven gesture production, applicable to virtual agents to enhance human-computer interaction. Specifically, we extend recent deep-learning-based, data-driven methods for speech-driven gesture generation by incorporating representation learning. Our model takes speech as input and produces gestures as output, in the form of a sequence of 3D coordinates. We provide an analysis of different representations for the input (speech) and the output (motion) of the network by both objective and subjective evaluations. We also analyse the importance of smoothing of the produced motion. Our results indicated that the proposed method improved on our baseline in terms of objective measures. For example, it better captured the motion dynamics and better matched the motion-speed distribution. Moreover, we performed user studies on two different datasets. The studies confirmed that our proposed method is perceived as more natural than the baseline, although the difference in the studies was eliminated by appropriate post-processing: hip-centering and smoothing. We conclude that it is important to take both feature representation, model architecture and post-processing into account when designing an automatic gesture-production method.
BibTeX
@article{kucherenko2021moving,
  author = {Taras Kucherenko and Dai Hasegawa and Naoshi Kaneko and Gustav Eje Henter and Hedvig Kjellström},
  title = {{Moving fast and slow: Analysis of representations and post-processing in speech-driven automatic gesture generation}},
  journal = {Int. J. Hum. Comput. Interact.},
  year = {2021},
  doi = {10.1080/10447318.2021.1883883},
}
Figure