Abstract
Gesticulator is a novel data-driven framework designed to generate continuous 3D co-speech gestures by integrating both acoustic (audio) and semantic (text) features. By leveraging an autoregressive deep learning architecture, the system simultaneously produces rhythmic "beat" gestures and meaningful "semantic" gestures. Experimental results demonstrate that Gesticulator significantly outperforms single-modality baselines in terms of human-likeness and alignment with speech.
Introduction
Human communication is a multimodal process where speech is naturally accompanied by spontaneous gesticulation. These gestures generally fall into two categories: beat gestures, which follow the rhythm and prosody of speech, and representational gestures, which convey specific meanings or shapes.
Traditional gesture generation systems typically rely on a single input modality. Audio-based systems excel at capturing rhythm but lack semantic understanding, while text-based systems can generate meaningful movements but often fail to synchronize with the speaker's actual intonation. Gesticulator bridges this gap by creating a unified model that respects both the how (acoustics) and the what (semantics) of spoken language.
Methodology
The Gesticulator framework employs a deep-learning-based autoregressive model to ensure motion continuity and multimodal integration.
Multimodal Input and Architecture
The model processes 64-dimensional log-power mel-spectrograms for audio and 768-dimensional BERT embeddings for text. A sliding window provides context by considering 0.5 seconds of past speech and 1.0 second of future speech.
Key architectural components include:
- Autoregressive Feedback: Previous pose predictions are fed back into the network to maintain physical plausibility and motion smoothness.
- FiLM Conditioning: Feature-wise Linear Modulation (FiLM) is used to adaptively scale and shift internal activations based on the current motion state.
- Annealed Training: To prevent the model from over-relying on pose history (and ignoring speech), the authors used a multi-stage training process that gradually introduced autoregressive feedback.
Datasets and Preprocessing
The system was trained using the Trinity College Dataset, which includes 244 minutes of high-quality 3D motion capture and audio. The authors enhanced this dataset by providing manual transcriptions and punctuation, creating a publicly available text-aligned version. Motion data was downsampled to 20 fps, with joint angles represented as exponential maps to ensure mathematical continuity.
Experimental Results and Insights
Ablation studies and subjective evaluations confirmed the importance of the multimodal approach.
The Role of Modalities
Removing either text or audio significantly degraded user perception. Semantics were found to be essential for "human-likeness," while audio was critical for "sync." Interestingly, the authors found that avoiding Principal Component Analysis (PCA) for dimensionality reduction preserved subtle, high-frequency variations that make movement feel more realistic.
Performance Metrics
In pairwise preference tests, the full Gesticulator model was overwhelmingly preferred over single-modality variants ($p < 0.0001$). While the model produces motion with lower "jerk" than ground truth—indicating a slight over-smoothing effect common in deterministic models—it consistently outperformed previous CNN-GAN benchmarks in speech-alignment.
Conclusion and Future Work
Gesticulator establishes a new baseline for semantically-aware gesture generation. However, several challenges remain. The current model is deterministic, meaning it produces the same gesture for the same input, whereas human movement is inherently stochastic. Additionally, the model's training on a single speaker limits its stylistic diversity. Future research will explore probabilistic models, such as Normalizing Flows, and larger, multi-speaker datasets to enhance the variety and naturalness of generated gestures.
Summary updated