X-Dancer: Expressive Music to Human Dance Video Generation

Zeyuan Chen, Hongyi Xu, Guoxian Song, You Xie, Chenxu Zhang, Xin Chen, Chao Wang, Di Chang, Linjie Luo

Abstract

X-Dancer is a novel zero-shot framework designed to generate high-fidelity, music-synchronized dance videos from a single static image. By integrating an autoregressive transformer for 2D pose generation with a diffusion-based animation pipeline, X-Dancer captures expressive full-body motions, including fine-grained hand and head movements. The system significantly outperforms existing state-of-the-art methods in both realism and musical alignment.

Introduction

Generating realistic human dance videos from music is a long-standing challenge in computer vision. Two primary hurdles have hindered progress:

  1. Data Scarcity: High-quality motion datasets typically rely on 3D skeletons (e.g., AIST++), which are expensive to produce and often lack detail in extremities like hands and faces.
  2. Synthesis Complexity: Maintaining a subject's identity and background consistency during high-velocity movements (jumps, rotations) while ensuring perfect synchronization with a musical beat is technically demanding.

Existing methods frequently produce jittery motions or fail to preserve the subject's likeness. X-Dancer addresses these issues by shifting the focus from limited 3D data to widely available 2D monocular videos, enabling the model to learn a broader vocabulary of movement.

Methodology

The X-Dancer framework employs a two-stage transformer-diffusion architecture to bridge the gap between audio features and visual synthesis.

Stage 1: Compositional Pose Generation

The model utilizes a Vector Quantized Variational Autoencoder (VQ-VAE) to decompose 2D human poses into discrete tokens. A multi-part scheme is employed, tokenizing five body regions (head, hands, upper/lower body) independently. A GPT-like transformer then predicts these tokens autoregressively, conditioned on musical features extracted via Jukebox and Librosa.

Stage 2: Implicit Motion Guidance

Unlike traditional approaches that render explicit "stick figure" skeleton maps, X-Dancer introduces a trainable motion decoder. This decoder takes 1D pose tokens and injects them directly into a Stable Diffusion 1.5 backbone using Adaptive Instance Normalization (AdaIN). This "implicit" guidance is fully differentiable and handles motion blur and occlusions more effectively than rigid maps.

The X-Dancer architecture, featuring a two-stage transformer-diffusion framework.

To ensure identity preservation, a separate "ReferenceNet" extracts features from the original static image, maintaining the dancer’s appearance throughout the sequence.

Key Insights

  • 2D Scalability: By modeling 2D poses, the researchers leveraged a dataset of 76,818 monocular recordings (~360 hours), nearly 50x larger than standard 3D dance sets.
  • Confidence-Aware Modeling: The pose representation includes keypoint confidence scores, allowing the transformer to learn realistic motion blur and handle temporary occlusions.
  • Implicit Guidance: Moving away from non-differentiable skeleton maps allows the diffusion model to adapt motion more naturally to the subject's specific body shape.

Experimental Results

X-Dancer was evaluated against top-tier baselines, including Hallo, Bailando, and EDGE.

Quantitative Performance

On an in-house dataset, X-Dancer achieved a Fréchet Video Distance (FVD) of 507.06, significantly outperforming Hallo (609.08). It also led in ID-SIM (0.5317), demonstrating superior identity preservation.

Qualitative Analysis and User Study

Qualitatively, X-Dancer generates fine-grained motions, such as finger movements and head tilts, which are often missed by competing models. In a blind user study, participants preferred X-Dancer 84.8% of the time across metrics including motion consistency and overall quality.

Qualitative comparison of X-Dancer against state-of-the-art baselines.

Conclusion and Future Work

While X-Dancer sets a new state-of-the-art, limitations remain. Extremely complex or "out-of-domain" poses can still result in rendering artifacts. Future work will focus on moving toward a fully end-to-end training setup and improving precision for technical dance styles like ballet.

Summary updated

BibTeX
@inproceedings{chen2025x,
  author = {Zeyuan Chen and Hongyi Xu and Guoxian Song and You Xie and Chenxu Zhang and Xin Chen and Chao Wang and Di Chang and Linjie Luo},
  title = {{X-Dancer: Expressive Music to Human Dance Video Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure