GCDance: Genre-Controlled 3D Full Body Dance Generation Driven By Music

Xinran Liu, Xu Dong, Diptesh Kanojia, Wenwu Wang, Zhenhua Feng

Abstract

Generating high-quality full-body dance sequences from music is a challenging task as it requires strict adherence to genre-specific choreography. Moreover, the generated sequences must be both physically realistic and precisely synchronized with the beats and rhythm of the music. To overcome these challenges, we propose GCDance, a classifier-free diffusion framework for generating genre-specific dance motions conditioned on both music and textual prompts. Specifically, our approach extracts music features by combining high-level pre-trained music foundation model features with hand-crafted features for multi-granularity feature fusion. To achieve genre controllability, we leverage CLIP to efficiently embed genre-based textual prompt representations at each time step within our dance generation pipeline. Our GCDance framework can generate diverse dance styles from the same piece of music while ensuring coherence with the rhythm and melody of the music. Extensive experimental results obtained on the FineDance dataset demonstrate that GCDance significantly outperforms the existing state-of-the-art approaches, which also achieve competitive results on the AIST++ dataset. Our ablation and inference time analysis demonstrate that GCDance provides an effective solution for high-quality music-driven dance generation.

Introduction

Generating dance from music is a complex cross-modal task that requires a model to satisfy three difficult criteria simultaneously: rhythmic alignment, physical realism, and stylistic consistency. While many existing models can match a beat, they often struggle with stylistic variety, typically producing a "generic" dance style regardless of the music's genre. Furthermore, many models ignore fine-grained details like hand and finger movements, which are essential for expressive choreography.

Comparison of GCDance with previous single-style models, demonstrating genre-controllability through text prompts.

GCDance addresses these limitations by introducing a novel diffusion-based framework that generates 3D full-body dance motions conditioned on both music and textual genre descriptions. Unlike previous models that are limited to a single style, GCDance allows users to specify genres or stylistic descriptions (e.g., "Jazz dance" or "Energetic dance with powerful freezes") to guide the generation process, resulting in motions that are both stylistically consistent and rhythmically aligned.

Methodology

The GCDance framework is built upon several key technical innovations designed to bridge the gap between audio, text, and motion.

Diffusion-Based Architecture and Multimodal Conditioning

Instead of using older autoregressive models that often "freeze" or drift over time, GCDance employs a diffusion process to iteratively refine noise into a coherent 4-second dance sequence. The model "understands" the input music through a combination of low-level features (STFT) and high-level semantic features from Wav2CLIP, a music foundation model. This multi-granularity fusion ensures the model captures both the raw rhythm and the overall "vibe" of the track.

Overview of the GCDance architecture, including feature extraction, reverse denoising, and the sampling process.

Genre-Control and Optimization

To achieve precise genre control, the model accepts text prompts which are embedded using CLIP. These signals are injected into the denoising process at every step using Feature-wise Linear Modulation (FiLM) layers. This allows the model to dynamically adapt the motion style based on the user's instruction.

Training such a complex model involves balancing multiple objectives, including motion quality, joint position, velocity, foot contact, and genre classification. To avoid the pitfalls of manual weight tuning, GCDance utilizes Nash MTL and Aligned MTL. These game-theory-inspired multi-task optimization strategies treat different losses like players in a bargaining game, finding an optimal update direction that satisfies all requirements without letting one objective dominate the others.

Full-Body Representation

Unlike standard 24-joint SMPL models, GCDance is trained on a 52-joint skeleton. This inclusion of intricate hand and finger movements is a significant step forward for realistic and expressive digital choreography.

Experimental Evaluation

The researchers evaluated GCDance on two primary benchmarks: FineDance (7.7 hours of 52-joint motion across 16 genres) and AIST++ (5.2 hours of 24-joint motion across 10 genres).

Key Results

GCDance achieved state-of-the-art (SOTA) performance across multiple metrics:

  • Motion Quality: On the FineDance dataset, GCDance-Nash outperformed the previous leader (EDGE) by 30.27% in hand FID and 55.58% in body FID, indicating significantly higher realism.
  • Physicality: The model achieved the best scores for Physical Body Contact (PBC), meaning the generated motions are more anatomically and physically grounded, avoiding common issues like feet "skating" across the floor.
  • User Preference: In a user study, participants preferred GCDance's results over baselines at least 63.26% of the time.
  • Efficiency: The model is highly efficient, generating a 4-second clip in just 0.22 seconds on a modern GPU, making it suitable for interactive applications.

Conclusion

GCDance represents a significant advancement in music-driven dance generation by successfully integrating textual genre control into a diffusion-based pipeline. By leveraging foundation models like Wav2CLIP and CLIP, and employing sophisticated multi-task optimization, it produces dances that are rhythmically precise and stylistically diverse.

While the model currently excels at genre-level control, future work will explore more granular attribute control (e.g., specifying intensity for specific limbs) and time-varying prompts to allow for stylistic transitions within a single piece of music. The ability to perform spatial and temporal editing (inpainting) further establishes GCDance as a versatile tool for digital animators and creators.

Summary updated

Abstract

Generating high-quality full-body dance sequences from music is a challenging task as it requires strict adherence to genre-specific choreography. Moreover, the generated sequences must be both physically realistic and precisely synchronized with the beats and rhythm of the music. To overcome these challenges, we propose GCDance, a classifier-free diffusion framework for generating genre-specific dance motions conditioned on both music and textual prompts. Specifically, our approach extracts music features by combining high-level pre-trained music foundation model features with hand-crafted features for multi-granularity feature fusion. To achieve genre controllability, we leverage CLIP to efficiently embed genre-based textual prompt representations at each time step within our dance generation pipeline. Our GCDance framework can generate diverse dance styles from the same piece of music while ensuring coherence with the rhythm and melody of the music. Extensive experimental results obtained on the FineDance dataset demonstrate that GCDance significantly outperforms the existing state-of-the-art approaches, which also achieve competitive results on the AIST++ dataset. Our ablation and inference time analysis demonstrate that GCDance provides an effective solution for high-quality music-driven dance generation.
BibTeX
@misc{liu2025gcdance,
  author = {Xinran Liu and Xu Dong and Diptesh Kanojia and Wenwu Wang and Zhenhua Feng},
  title = {{GCDance: Genre-Controlled 3D Full Body Dance Generation Driven By Music}},
  year = {2025},
  eprint = {2502.18309},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.18309},
}
Figure