Realistic-Gesture: Co-Speech Gesture Video Generation through Semantic-aware Gesture Representation

Pinxin Liu, Pengfei Zhang, Hyeongwoo Kim, Pablo Garrido, Ari Shapiro, Kyle OlszewskiOpenReview 2025 · Submission to ICLR 2025 · Rejected · OpenReview

Abstract

Co-speech gesture generation is crucial for creating lifelike avatars and enhancing human-computer interactions by synchronizing gestures with speech in computer vision. Despite recent advancements, existing methods often struggle with accurately aligning gesture motions with speech signals and achieving pixel-level realism. To address these challenges, we introduce Realistic-Gesture, a groundbreaking framework that transforms co-speech gesture video generation through three innovative components: (1) a speech-aware gesture tokenization that incorporate speech context into motion pattern representation, (2) a mask gesture generator that learns to map audio signals to gestures by predicting masked motion tokens, enabling bidirectional contextually relevant gesture synthesis and editing, and (3) a structure-aware refinement module that employs differentiable edge connection to link gesture keypoints to improve video generation. Our extensive experiments demonstrate that Realistic-Gesture not only produces highly realistic and speech-aligned gesture videos but also supports long-sequence generation and video gesture editing applications.
BibTeX
@misc{liu2025realistic,
  author = {Pinxin Liu and Pengfei Zhang and Hyeongwoo Kim and Pablo Garrido and Ari Shapiro and Kyle Olszewski},
  title = {{Realistic-Gesture: Co-Speech Gesture Video Generation through Semantic-aware Gesture Representation}},
  year = {2025},
  url = {https://openreview.net/forum?id=EXsiGFkwV6},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}

Loading related content...

Figure