Abstract
Co-speech gesture generation is a critical component for creating realistic digital humans, yet models often struggle with the "boringness" problem—generating only common, repetitive motions while ignoring rare, expressive ones. This paper introduces AISLe (Adversarial Importance Sampled Learning), a training paradigm that leverages a GAN discriminator to perform importance sampling, effectively addressing the long-tail distribution of human gestures. Paired with a Multimodal Multi-Scale Transformer for precise sub-word alignment, the framework achieves state-of-the-art results in generating diverse and well-timed 2D gestures.
Introduction
When we communicate, our hands perform significant "heavy lifting" through pointing, shrugging, and rhythmic emphasis. Replicating this co-speech gesture behavior for AI-driven avatars remains a formidable challenge. Most existing models suffer from a lack of diversity, frequently collapsing into the most common, repetitive motions (such as rhythmic "beat" gestures) while failing to produce the rare, expressive "iconic" gestures that characterize human interaction.
Researchers from Carnegie Mellon University address this in their EMNLP 2020 paper by introducing a framework designed to ensure that no gesture—regardless of its rarity—is neglected during the learning process.
The Challenges of Gesture Synthesis
Generating gestures from speech is complicated by two primary factors:
- The Long-Tail Problem: Gesture data is inherently skewed. Humans spend the majority of their time making simple rhythmic movements. Standard training objectives often cause models to "collapse" toward these high-frequency samples to minimize error, ignoring meaningful but rare gestures.
- Granularity Mismatch: Language consists of discrete words, whereas audio and gestures are continuous signals occurring at a "sub-word" level. A gesture might be triggered by a specific intonation or pitch change that does not align perfectly with word boundaries.
Methodology: AISLe and MMS-Transformer
The proposed framework introduces two major technical innovations to balance precision (accuracy relative to speech) with coverage (diversity of gestures).
Adversarial Importance Sampled Learning (AISLe)
AISLe is the core mechanism for ensuring diversity. In a standard Generative Adversarial Network (GAN), a generator attempts to fool a discriminator. AISLe repurposes the discriminator's output to calculate "importance weights." As the model masters common gestures, the discriminator becomes better at identifying them. AISLe uses this information to shift training focus toward "long-tail" samples that the model has not yet mastered, forcing the generator to learn rare gestures without degrading the quality of common ones.
Multimodal Multi-Scale Transformer (MMS-Transformer)
To address the alignment problem, the authors utilize a cross-attention mechanism. Rather than forcing a rigid one-to-one mapping between words and audio frames, the transformer allows the model to learn sub-word relationships. By using audio features as "queries" to look up linguistic information, the model achieves significantly finer timing in generated motions, aligning gestures with the prosody of speech.
Addressing the Long-Tail Distribution
The effectiveness of AISLe is best understood by comparing it to traditional sampling methods. While "No Resampling" leaves the model biased toward common gestures, and "Static Resampling" might over-correct or lose nuance, AISLe dynamically adjusts to the model's current learning state.
Experimental Setup and Results
The model was trained on an extended version of the PATS (Pose, Audio, Transcripts, and Style) dataset, covering over 250 hours of video from 25 diverse speakers. The researchers added automatically extracted, time-aligned transcripts to facilitate multimodal training.
Evaluation against state-of-the-art models like Speech2Gesture and Gesticulator revealed significant improvements:
- Human Preference: In user studies, the proposed model was significantly preferred for Naturalness, Expressivity, Relevance, and Timing.
- Coverage (FID): The Fréchet Inception Distance (FID) improved from 68.1 (Speech2Gesture) to 27.8, indicating a much closer match to the real data distribution.
- Precision (PCK): The model maintained high Probability of Correct Keypoints (PCK), demonstrating that increased diversity does not come at the cost of accuracy.
Conclusion
By framing the GAN discriminator as a density estimator, the AISLe framework provides a robust solution to the long-tail problem in gesture synthesis. While future work remains in transitioning to 3D poses and capturing fine-grained finger movements, this research marks a significant step toward more expressive and human-like digital avatars.
Summary updated