Gesticulator: A framework for semantically-aware speech-driven gesture generation

Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, Hedvig Kjellström

Abstract

Gesticulator is a novel data-driven framework designed to generate continuous 3D co-speech gestures by integrating both acoustic (audio) and semantic (text) features. By leveraging an autoregressive deep learning architecture, the system simultaneously produces rhythmic "beat" gestures and meaningful "semantic" gestures. Experimental results demonstrate that Gesticulator significantly outperforms single-modality baselines in terms of human-likeness and alignment with speech.

Introduction

Human communication is a multimodal process where speech is naturally accompanied by spontaneous gesticulation. These gestures generally fall into two categories: beat gestures, which follow the rhythm and prosody of speech, and representational gestures, which convey specific meanings or shapes.

Traditional gesture generation systems typically rely on a single input modality. Audio-based systems excel at capturing rhythm but lack semantic understanding, while text-based systems can generate meaningful movements but often fail to synchronize with the speaker's actual intonation. Gesticulator bridges this gap by creating a unified model that respects both the how (acoustics) and the what (semantics) of spoken language.

Methodology

The Gesticulator framework employs a deep-learning-based autoregressive model to ensure motion continuity and multimodal integration.

Multimodal Input and Architecture

The model processes 64-dimensional log-power mel-spectrograms for audio and 768-dimensional BERT embeddings for text. A sliding window provides context by considering 0.5 seconds of past speech and 1.0 second of future speech.

The Gesticulator architecture integrates text and audio features using FiLM conditioning and autoregressive feedback.

Key architectural components include:

  • Autoregressive Feedback: Previous pose predictions are fed back into the network to maintain physical plausibility and motion smoothness.
  • FiLM Conditioning: Feature-wise Linear Modulation (FiLM) is used to adaptively scale and shift internal activations based on the current motion state.
  • Annealed Training: To prevent the model from over-relying on pose history (and ignoring speech), the authors used a multi-stage training process that gradually introduced autoregressive feedback.

Datasets and Preprocessing

The system was trained using the Trinity College Dataset, which includes 244 minutes of high-quality 3D motion capture and audio. The authors enhanced this dataset by providing manual transcriptions and punctuation, creating a publicly available text-aligned version. Motion data was downsampled to 20 fps, with joint angles represented as exponential maps to ensure mathematical continuity.

Experimental Results and Insights

Ablation studies and subjective evaluations confirmed the importance of the multimodal approach.

The Role of Modalities

Removing either text or audio significantly degraded user perception. Semantics were found to be essential for "human-likeness," while audio was critical for "sync." Interestingly, the authors found that avoiding Principal Component Analysis (PCA) for dimensionality reduction preserved subtle, high-frequency variations that make movement feel more realistic.

Velocity distribution comparisons across different architectures (top) and input modalities (bottom).

Performance Metrics

In pairwise preference tests, the full Gesticulator model was overwhelmingly preferred over single-modality variants ($p < 0.0001$). While the model produces motion with lower "jerk" than ground truth—indicating a slight over-smoothing effect common in deterministic models—it consistently outperformed previous CNN-GAN benchmarks in speech-alignment.

Conclusion and Future Work

Gesticulator establishes a new baseline for semantically-aware gesture generation. However, several challenges remain. The current model is deterministic, meaning it produces the same gesture for the same input, whereas human movement is inherently stochastic. Additionally, the model's training on a single speaker limits its stylistic diversity. Future research will explore probabilistic models, such as Normalizing Flows, and larger, multi-speaker datasets to enhance the variety and naturalness of generated gestures.

Summary updated

BibTeX
@inproceedings{kucherenko2020gesticulator,
  author = {Taras Kucherenko and Patrik Jonell and Sanne van Waveren and Gustav Eje Henter and Simon Alexandersson and Iolanda Leite and Hedvig Kjellström},
  title = {{Gesticulator: A framework for semantically-aware speech-driven gesture generation}},
  booktitle = {ICMI},
  year = {2020},
  doi = {10.1145/3382507.3418815},
}
Figure