Abstract
Dance plays an important role as an artistic form and expression in human culture, yet the creation of dance remains a challenging task. Most dance generation methods primarily rely solely on music, seldom taking into consideration intrinsic attributes such as music style or genre. In this work, we introduce Flexible Dance Generation with Style Description Prompts (DGSDP), a diffusion-based framework suitable for diversified tasks of dance generation by fully leveraging the semantics of music style. The core component of this framework is Music-Conditioned Style-Aware Diffusion (MCSAD), which comprises a Transformer-based network and a music Style Modulation module. The MCSAD seemly integrates music conditions and style description prompts into the dance generation framework, ensuring that generated dances are consistent with the music content and style. To facilitate flexible dance generation and accommodate different tasks, a spatial-temporal masking strategy is effectively applied in the backward diffusion process. The proposed framework successfully generates realistic dance sequences that are accurately aligned with music for a variety of tasks such as long-term generation, dance in-betweening, dance inpainting, and etc. Code is available on Github: https://github.com/mucunzhuzhu/DGSDP.
Introduction
While AI has become adept at generating motion that matches the "beat" of music, it often fails at the "soul" of dance—the style. Most existing models treat dance as a generic sequence of poses, ignoring the nuanced differences between various genres, such as the fluid movements of ballet versus the sharp hits of locking. Furthermore, "controllability" is a major hurdle for professional creative workflows. If a choreographer wants an AI to generate a dance where the character moves along a specific path or keeps their lower body still while improvising with their arms, most current models struggle as they lack surgical precision.
Methods: Style-Guided Motion Diffusion
The researchers introduce Style-Guided Motion Diffusion (SGMD), a framework that allows users to control AI dance generation using both natural language style descriptions and spatial-temporal constraints. The architecture introduces two primary innovations:
- Style Modulation (SM) Module: Instead of just feeding the model a genre label, the authors use a lightweight layer to inject "Style Prompts." These prompts are processed via a Transformer-based decoder. The SM module normalizes the input features and then scales them using style embeddings, ensuring the "vibe" of the dance is baked into the motion without distorting the underlying musical alignment.
- Spatial-Temporal Masking: To enable control, the authors implement a 2D masking strategy (Frames × Joints). During the denoising process, the model can be "forced" to keep certain joints or certain time-slices fixed to a user-provided "known" sequence. This allows for tasks like in-painting or trajectory-based generation.
Key Insights and Results
One of the most significant findings is that Style Description Prompts (detailed sentences generated by GPT-3) perform significantly better than simple one-hot category encodings. The richer semantic information helps the model better map musical "mood" to physical "motion." Additionally, the use of Jukebox for audio feature extraction provided the best features for aligning motion to the beat.
The SGMD model outperformed the state-of-the-art baseline (EDGE) across several metrics on the AIST++ dataset:
- Beat Alignment: SGMD achieved a score of 0.31, compared to EDGE’s 0.26, indicating much tighter synchronization with the music.
- Physical Plausibility: The model showed lower Physical Foot Contact (PFC) errors in several controllable tasks, meaning the feet "slid" less on the floor.
- User Preference: In a user study, 60% of participants preferred SGMD's generated dances over the baseline, citing better diversity and stylistic consistency.
Conclusion
SGMD represents a significant step forward in making AI-generated choreography both more artistic and functional. While current limitations include a potential disconnect between rhythm and text prompts in conflicting scenarios, the framework's zero-shot task versatility makes it a powerful tool for creators. Future work may explore autoregressive frameworks to allow for the generation of infinitely long, seamless dance sequences.
Reference: Wang, H., et al. (2024). Controllable Dance Generation with Style-Guided Motion Diffusion. arXiv:2406.07871v3.
Summary updated