Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers

Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Hideki Koike

Introduction

Generating realistic video of a person speaking involves more than just synchronizing lip movements to audio. Natural communication relies heavily on co-speech gestures—the rhythmic movements of the hands and body that accompany speech. While "talking head" models have made significant strides in facial animation, they often fail to produce convincing body movements, frequently resulting in blurry hands or awkward, rigid postures.

Cosh-DiT introduces a novel two-stage framework designed to bridge the gap between audio-driven motion generation and high-fidelity video synthesis. By leveraging Diffusion Transformers (DiTs) and a unique hybrid motion representation, the system generates realistic, temporally consistent videos of avatars gesturing naturally to speech.

The Challenge of Co-Speech Gestures

Existing co-speech gesture models typically fall into two camps: those that generate 3D skeletons/meshes and those that attempt to generate video directly from audio. Both approaches face significant hurdles. Full 3D body reconstruction is often highly unstable because most training data only captures the upper body, leading to artifacts in the neck and shoulders.

Pure 3D body reconstructions often exhibit instability and artifacts (highlighted in red) when extracted from standard talking-head videos.

Conversely, models that bypass intermediate skeletons often struggle with spatial coherence, producing "blurry hands" and flickering movements. The core challenge lies in finding a representation that provides enough geometric constraint for realistic hands without the instability of a full 3D body model.

Methodology: The Cosh-DiT Framework

The authors propose a two-stage system built entirely on Diffusion Transformers, utilizing a "Goldilocks" approach to motion representation.

1. Hybrid Motion Representation

Instead of choosing between 2D and 3D, Cosh-DiT uses both. It employs 2D skeletons for the upper body—which are stable and easy to extract from video—and 3D parametric models (MANO) for the hands to preserve finger shape and depth. A Geometric-Aware Alignment module then projects the 3D hands onto the 2D plane to ensure seamless connection at the wrists.

The Cosh-DiT hybrid representation combines 2D body segments with 3D parametric hand models to balance stability and geometric detail.

2. Two-Stage Diffusion Pipeline

  • Stage 1: Audio-to-Motion (Cosh-DiT-A): The hybrid motion is compressed into discrete tokens via a VQ-VAE. A discrete Diffusion Transformer then predicts these tokens based on audio features (Wav2Vec2 and BEAT). Using discrete diffusion helps avoid the compounding errors common in autoregressive models.
  • Stage 2: Motion-to-Video (Cosh-DiT-V): The predicted motion is rendered into a visual guide. A continuous video Diffusion Transformer then synthesizes the final video using a reference image. A novel "Appearance Motion" (AM) block processes identity and motion in separate streams, merging them via joint attention to preserve fine textures like clothing while maintaining fluid movement.

Key Results and Insights

Cosh-DiT was evaluated in a zero-shot setting against state-of-the-art pipelines, including TalkShow and S2G-MDD. The model significantly outperformed baselines across several key metrics:

  • Temporal Consistency: Achieved a Frechet Video Distance (FVD) of 881.83, a substantial improvement over the next best baseline (1118.94).
  • Image Fidelity: Scored highest in structural similarity (SSIM: 0.6814) and lowest in image distortion (FID: 79.78).

Qualitatively, Cosh-DiT produces sharp, structurally sound hands and maintains consistent facial identity, whereas competing methods often generate mangled fingers or rigid, unrealistic body poses.

Conclusion and Future Work

Cosh-DiT demonstrates that decoupling identity from motion and utilizing a hybrid 2D/3D representation can solve long-standing issues in co-speech gesture synthesis. While the model currently faces challenges with complex backgrounds and intricate hand interactions (like crossed fingers), it sets a new benchmark for creating realistic virtual anchors. Future research will focus on scaling the model to handle more diverse, real-world environments beyond centralized avatars.

Summary updated

Abstract

Co-speech gesture video synthesis is a challenging task that requires both probabilistic modeling of human gestures and the synthesis of realistic images that align with the rhythmic nuances of speech. To address these challenges, we propose Cosh-DiT, a Co-speech gesture video system with hybrid Diffusion Transformers that perform audio-to-motion and motion-to-video synthesis using discrete and continuous diffusion modeling, respectively. First, we introduce an audio Diffusion Transformer (Cosh-DiT-A) to synthesize expressive gesture dynamics synchronized with speech rhythms. To capture upper body, facial, and hand movement priors, we employ vector-quantized variational autoencoders (VQ-VAEs) to jointly learn their dependencies within a discrete latent space. Then, for realistic video synthesis conditioned on the generated speech-driven motion, we design a visual Diffusion Transformer (Cosh-DiT-V) that effectively integrates spatial and temporal contexts. Extensive experiments demonstrate that our framework consistently generates lifelike videos with expressive facial expressions and natural, smooth gestures that align seamlessly with speech.
BibTeX
@article{sun2026cosh,
  author = {Yasheng Sun and Zhiliang Xu and Hang Zhou and Jiazhi Guan and Quanwei Yang and Kaisiyuan Wang and Borong Liang and Yingying Li and Haocheng Feng and Jingdong Wang and Ziwei Liu and Hideki Koike},
  title = {{Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers}},
  journal = {Int. J. Comput. Vis.},
  year = {2026},
  doi = {10.1007/s11263-026-02752-z},
}
Figure