No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures

Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, Louis-Philippe Morency

Abstract

Co-speech gesture generation is a critical component for creating realistic digital humans, yet models often struggle with the "boringness" problem—generating only common, repetitive motions while ignoring rare, expressive ones. This paper introduces AISLe (Adversarial Importance Sampled Learning), a training paradigm that leverages a GAN discriminator to perform importance sampling, effectively addressing the long-tail distribution of human gestures. Paired with a Multimodal Multi-Scale Transformer for precise sub-word alignment, the framework achieves state-of-the-art results in generating diverse and well-timed 2D gestures.

Introduction

When we communicate, our hands perform significant "heavy lifting" through pointing, shrugging, and rhythmic emphasis. Replicating this co-speech gesture behavior for AI-driven avatars remains a formidable challenge. Most existing models suffer from a lack of diversity, frequently collapsing into the most common, repetitive motions (such as rhythmic "beat" gestures) while failing to produce the rare, expressive "iconic" gestures that characterize human interaction.

Researchers from Carnegie Mellon University address this in their EMNLP 2020 paper by introducing a framework designed to ensure that no gesture—regardless of its rarity—is neglected during the learning process.

The Challenges of Gesture Synthesis

Generating gestures from speech is complicated by two primary factors:

  1. The Long-Tail Problem: Gesture data is inherently skewed. Humans spend the majority of their time making simple rhythmic movements. Standard training objectives often cause models to "collapse" toward these high-frequency samples to minimize error, ignoring meaningful but rare gestures.
  2. Granularity Mismatch: Language consists of discrete words, whereas audio and gestures are continuous signals occurring at a "sub-word" level. A gesture might be triggered by a specific intonation or pitch change that does not align perfectly with word boundaries.

Methodology: AISLe and MMS-Transformer

The proposed framework introduces two major technical innovations to balance precision (accuracy relative to speech) with coverage (diversity of gestures).

Adversarial Importance Sampled Learning (AISLe)

AISLe is the core mechanism for ensuring diversity. In a standard Generative Adversarial Network (GAN), a generator attempts to fool a discriminator. AISLe repurposes the discriminator's output to calculate "importance weights." As the model masters common gestures, the discriminator becomes better at identifying them. AISLe uses this information to shift training focus toward "long-tail" samples that the model has not yet mastered, forcing the generator to learn rare gestures without degrading the quality of common ones.

The AISLe framework architecture, integrating multimodal attention, a GAN-based generator-discriminator pair, and an importance sampling loop to update resampling weights.

Multimodal Multi-Scale Transformer (MMS-Transformer)

To address the alignment problem, the authors utilize a cross-attention mechanism. Rather than forcing a rigid one-to-one mapping between words and audio frames, the transformer allows the model to learn sub-word relationships. By using audio features as "queries" to look up linguistic information, the model achieves significantly finer timing in generated motions, aligning gestures with the prosody of speech.

Addressing the Long-Tail Distribution

The effectiveness of AISLe is best understood by comparing it to traditional sampling methods. While "No Resampling" leaves the model biased toward common gestures, and "Static Resampling" might over-correct or lose nuance, AISLe dynamically adjusts to the model's current learning state.

Comparison of sampling strategies: (a) No Resampling, (b) Static Resampling, and (c) AISLe, which dynamically adjusts weights to cover the long-tail distribution of gestures.

Experimental Setup and Results

The model was trained on an extended version of the PATS (Pose, Audio, Transcripts, and Style) dataset, covering over 250 hours of video from 25 diverse speakers. The researchers added automatically extracted, time-aligned transcripts to facilitate multimodal training.

Evaluation against state-of-the-art models like Speech2Gesture and Gesticulator revealed significant improvements:

  • Human Preference: In user studies, the proposed model was significantly preferred for Naturalness, Expressivity, Relevance, and Timing.
  • Coverage (FID): The Fréchet Inception Distance (FID) improved from 68.1 (Speech2Gesture) to 27.8, indicating a much closer match to the real data distribution.
  • Precision (PCK): The model maintained high Probability of Correct Keypoints (PCK), demonstrating that increased diversity does not come at the cost of accuracy.

Conclusion

By framing the GAN discriminator as a density estimator, the AISLe framework provides a robust solution to the long-tail problem in gesture synthesis. While future work remains in transitioning to 3D poses and capturing fine-grained finger movements, this research marks a significant step toward more expressive and human-like digital avatars.

Summary updated

Abstract

We study relationships between spoken language and co-speech gestures in context of two key challenges. First, distributions of text and gestures are inherently skewed making it important to model the long tail. Second, gesture predictions are made at a subword level, making it important to learn relationships between language and acoustic cues. We introduce AISLe, which combines adversarial learning with importance sampling to strike a balance between precision and coverage. We propose the use of a multimodal multiscale attention block to perform subword alignment without the need of explicit alignment between language and acoustic cues. Finally, to empirically study the importance of language in this task, we extend the dataset proposed in Ahuja et al. (2020) with automatically extracted transcripts for audio signals. We substantiate the effectiveness of our approach through large-scale quantitative and user studies, which show that our proposed methodology significantly outperforms previous state-of-the-art approaches for gesture generation. Link to code, data and videos: https://github.com/chahuja/aisle
BibTeX
@inproceedings{ahuja2020no,
  author = {Chaitanya Ahuja and Dong Won Lee and Ryo Ishii and Louis-Philippe Morency},
  title = {{No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.170},
}
Figure