MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generation

Siyuan Wang, Jiawei Liu, Wei Wang, Yeying Jin, Jinsong Du, Zhi Han

TL;DR

This paper introduces MMGT, a novel two-stage network for generating realistic co-speech gesture videos from audio and a single reference image. MMGT addresses limitations in existing methods by effectively capturing large body movements and fine-grained facial and hand gestures through a motion mask-guided approach, leading to improved video quality and synchronization.

What Problem Does This Paper Address?

Generating realistic videos of people speaking and gesturing in sync with their audio is a challenging task. While existing methods can produce co-speech videos, they often struggle with:

  • Capturing large, dynamic movements: Relying solely on audio can lead to artifacts and distortions, especially for significant gestures and head movements.
  • Balancing quality and practicality: Some high-quality methods require additional "a priori" information (like pre-recorded motion sequences), which limits their real-world applicability.
  • Fine-grained control and region-specific details: Generating synchronized facial expressions and gestures with accurate texture and motion details simultaneously is difficult.

This paper aims to bridge this gap by proposing a method that uses only audio and a reference image to generate high-quality, synchronized co-speech gesture videos, overcoming the limitations of previous approaches.

Methods

The proposed MMGT framework operates in two stages:

  1. Stage I: Spatial Mask-Guided Audio Pose Generation (SMGA):
    • This stage takes audio and an initial pose (extracted from the reference image) to generate a high-quality pose video and a corresponding "motion mask."
    • The motion mask highlights regions of significant movement, such as the face and hands, aiding in capturing large gestures.
  2. The SMGA network uses self-attention and cross-attention mechanisms to correlate motion features and audio, with a FiLM layer for dynamic adjustment.
  3. Stage II: Motion Masked Hierarchical Audio Attention (MM-HAA) integrated into a Stabilized Diffusion Video Generation model:
    • This stage refines the generated motion features using a Denoising UNet.
  4. The core innovation here is the MM-HAA module, which dynamically aligns audio features with motion masks (for face, gestures, and background) using cross-attention. This allows for adaptive attention to specific semantic regions, enhancing fine-grained details.
  5. The model leverages spatial attention for texture, temporal attention for smooth transitions, and the MM-HAA for audio-motion alignment, ensuring temporal and spatial consistency.

Datasets

  • PATS dataset: Used for training and evaluation. It contains 84,000 clips from 25 speakers, totaling 251 hours. The authors focused on a subset of four speakers, extracting 4,800 training clips and 2,048 test clips.

Key Insights and Ideas

  • Motion Masks are Crucial: The paper demonstrates that explicitly generating and using motion masks (highlighting key moving regions) is vital for accurately capturing large gestures and improving overall video quality, especially in contrast to static masks.
  • Two-Stage Approach for Complementary Strengths: Separating motion feature generation (SMGA) from detail refinement (Stage II diffusion model) allows each stage to specialize. SMGA excels at capturing broad movements, while the second stage focuses on fine-tuning details guided by the motion masks and audio.
  • Hierarchical Audio Attention for Region-Specific Refinement: The MM-HAA module's ability to adaptively focus on different body parts (face, gestures, background) based on motion masks and audio features is key to achieving high-fidelity, synchronized results.
  • Practicality through Minimal Input: The framework's ability to generate high-quality videos using only audio and a single reference image significantly enhances its practical applicability compared to methods requiring more complex inputs.

Key Results

  • Quantitative Improvements: MMGT significantly outperforms baseline models (MM-Diffusion, EDGE + Animate-anyone, EDGE + Hallo) across various metrics, including:
  • Gesture Quality: Lower Frechet Gesture Distance (FGD) (6.02 vs. 6.72 for baselines).
  • Video Quality: Lower Frechet Video Distance (FVD) (230.89 vs. 398.10 for EDGE+Hallo) and improved PSNR, SSIM, and LPIPS.
  • Lip Synchronization: Superior Syn-C and Syn-D scores, indicating better alignment with audio.
  • Qualitative Improvements: Visual comparisons show MMGT generates more natural, realistic, and synchronized co-speech gesture videos with fewer artifacts, better hand motion accuracy, and more consistent identity preservation compared to existing methods.
  • Ablation Studies: The studies confirm the importance of each component: SMGA for robust lip sync and motion quality, motion-specific facial loss for natural expression, motion masks for temporal coherence and detail, and audio guidance for realistic lip movements.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges potential misuse for creating deceptive deepfakes and privacy violations, highlighting the need for ethical guidelines and robust data governance.
  • While MMGT excels, the paper doesn't explicitly detail limitations in terms of computational cost or specific failure cases beyond what's implied by ablation studies (e.g., what happens with very complex or unusual gestures).
  • Future Work:
    • The authors suggest further research into ethical considerations and mitigation strategies for potential misuse of the technology.
  • The framework's flexibility is noted, hinting at future work in pose-driven generation and joint driving with both pose and audio, offering greater control.

Summary updated

BibTeX
@article{wang2026mmgt,
  author = {Siyuan Wang and Jiawei Liu and Wei Wang and Yeying Jin and Jinsong Du and Zhi Han},
  title = {{MMGT: Motion Mask Guided Two-Stage Network for Co-Speech Gesture Video Generation}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2026},
  doi = {10.1109/tcsvt.2025.3604109},
}
Figure