Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers

Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Koike Hideki

TL;DR

Cosh-DiT is a novel two-stage framework that synthesizes highly realistic co-speech gesture videos from audio and a single reference image. By combining a discrete audio-to-motion Diffusion Transformer (Cosh-DiT-A) with a continuous motion-to-video visual Diffusion Transformer (Cosh-DiT-V), the system generates lifelike, rhythmically synchronized upper-body movements and highly detailed hand gestures without the cumulative errors typical of autoregressive models.

What Problem Does This Paper Address?

Generating a realistic video of a talking person—complete with natural hand gestures and body language that match the rhythm and semantics of speech—is a major open challenge in digital human creation.

Existing methods typically hit a wall when choosing an intermediate motion representation to guide the video generation. Full 3D mesh reconstruction (like SMPL-X) tends to be highly unstable for standard upper-body-only videos where legs and feet are hidden. On the other hand, relying purely on 2D skeletons fails to capture the depth, occlusions, and intricate details required for complex hand gestures and facial expressions. Furthermore, standard video generators often struggle to render fine-grained details, leading to blurry hands, distorted faces, or rigid, unnatural movements.

Comparison of 3D, 2D, and Hybrid motion representations.

Methods

Cosh-DiT splits the problem into two distinct stages using a hybrid motion representation and specialized Diffusion Transformers (DiTs):

  1. Hybrid Motion Representation & Alignment: The authors combine the best of both worlds: robust 2D upper-body skeletons (via DWPose) and parametric 3D MANO hand models. To seamlessly merge them, a Geometric-Aware Alignment Module dynamically optimizes the 3D hand positions on the fly using perspective projection constraints, ensuring the hands perfectly anchor to the 2D wrists.
  2. Audio-to-Motion Synthesis (Cosh-DiT-A): Instead of generating motion step-by-step using autoregressive models (which suffer from drift and error accumulation), Cosh-DiT-A uses a discrete diffusion approach. A VQ-VAE compresses the hybrid motion into discrete token indices. The transformer then iteratively denoises masked tokens based on speech features extracted from BEAT and Wav2Vec2 models.
  3. Motion-to-Video Synthesis (Cosh-DiT-V): This stage uses a continuous visual Diffusion Transformer to turn the rendered motion sequences, a reference identity image, and previous context frames into the final video.

Instead of processing identity and motion together in a single block, the authors introduce a stacked Appearance Motion (AM)-DiT block architecture. This design decouples the processing: an Appearance Block first integrates identity textures into the motion flow, and a Motion Block subsequently refens details against previous frames to guarantee temporal continuity.

The two-stage Cosh-DiT pipeline architecture.

Datasets

  • Internal Dataset: The authors collected a diverse dataset consisting of 200 hours of video footage featuring centralized 2D speaking avatars, encompassing 921 distinct subjects.
  • PATs Dataset: A public benchmark used to evaluate the model's generalization capabilities.
  • Evaluation Setup: Testing was strictly performed in a zero-shot setting, meaning none of the test identities were seen by the model during training.

Key Insights and Ideas

  • The Hybrid Sweet Spot: Blending 2D body skeletons with 3D hand meshes provides an ideal intermediate representation. It bypasses the instability of full-body 3D tracking in cropped videos while retaining the geometric constraints needed to prevent finger distortion.
  • Discrete Diffusion for Abstract Planning: Operating diffusion directly on discrete token indices (via VQ-VAE) allows the model to consider the entire temporal sequence globally, eliminating the uni-directional bias of autoregressive generation.
  • Decoupling Texture and Dynamics: Forcing a neural network to learn spatial identity textures and temporal motion dynamics simultaneously often degrades fine details. Separating them sequentially within the AM-DiT blocks drastically improves the clarity of hands and facial expressions.

Key Results

Cosh-DiT significantly outperformed state-of-the-art baselines (including combinations of TalkShow, ProbTalk, and EMAGE with video generators like ControlNeXt and MimicMotion).

Quantitative Performance

On the authors' dataset, Cosh-DiT achieved:

  • An FID score of 77.51 (outperforming the next best baseline, PT-CX, which scored 93.56), indicating superior image quality.
  • A Fréchet Video Distance (FVD) of 1118.94 (compared to 1342.13 for the best baseline), demonstrating vastly superior temporal coherence.
  • A hand detection confidence score (Conf-hand) of 0.8937, proving that the generated hands are structurally accurate and clear enough for off-the-shelf detectors to easily track.

Qualitative Performance

Visual evaluations show that Cosh-DiT successfully avoids the severe facial warping and hand artifacts common in other diffusion-based video animators, producing smooth, natural gestures that map convincingly to the speaker's voice.

Qualitative comparison of Cosh-DiT against state-of-the-art baselines.

Limitations and Future Work

While Cosh-DiT marks a notable advancement, the authors acknowledge a few limitations:

  • Unstable Backgrounds: The model's synthesis quality can degrade if the reference video contains complex or moving backgrounds.
  • Intricate Hand Interactions: Highly complex gestures, such as crossing fingers or interlocking hands, still pose difficulties for the underlying 3D hand representation.
  • Future Directions: The authors suggest that scaling up the framework with larger, scale-pre-trained visual DiT models could help resolve background artifacts and generalize better to diverse, real-world environments.

Summary updated

Abstract

Co-speech gesture video synthesis is a challenging task that requires both probabilistic modeling of human gestures and the synthesis of realistic images that align with the rhythmic nuances of speech. To address these challenges, we propose Cosh-DiT, a Co-speech gesture video system with hybrid Diffusion Transformers that perform audio-to-motion and motion-to-video synthesis using discrete and continuous diffusion modeling, respectively. First, we introduce an audio Diffusion Transformer (Cosh-DiT-A) to synthesize expressive gesture dynamics synchronized with speech rhythms. To capture upper body, facial, and hand movement priors, we employ vector-quantized variational autoencoders (VQ-VAEs) to jointly learn their dependencies within a discrete latent space. Then, for realistic video synthesis conditioned on the generated speech-driven motion, we design a visual Diffusion Transformer (Cosh-DiT-V) that effectively integrates spatial and temporal contexts. Extensive experiments demonstrate that our framework consistently generates lifelike videos with expressive facial expressions and natural, smooth gestures that align seamlessly with speech.
BibTeX
@article{sun2026cosh,
  author = {Yasheng Sun and Zhiliang Xu and Hang Zhou and Jiazhi Guan and Quanwei Yang and Kaisiyuan Wang and Borong Liang and Yingying Li and Haocheng Feng and Jingdong Wang and Ziwei Liu and Koike Hideki},
  title = {{Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers}},
  journal = {Int. J. Comput. Vis.},
  year = {2026},
  doi = {10.1007/s11263-026-02752-z},
}
Figure