Mining and Applying Composition Knowledge of Dance Moves for Style-Concentrated Dance Generation

Xinjian Zhang, Su Yang, Yi Xu, Weishan Zhang, Longwen GaoAAAI 2023 · DOI

Introduction

While artificial intelligence has made significant strides in generating movement from musical inputs, most existing models suffer from a fundamental "style-blindness." These systems typically treat music as the sole determinant of motion, overlooking the reality that a single piece of music can be interpreted through various choreographic lenses—from the sharp, robotic isolations of Popping to the fluid, high-energy sequences of Anime-style dance.

Without explicit stylized knowledge, current generative models often produce movements that are either overly rigid, repetitive, or a "muddled" hybrid of styles that lack artistic coherence. Furthermore, users have historically had minimal control over the specific "flavor" of the generated choreography. This research introduces a novel framework designed to bridge this gap by teaching machines to recognize, extract, and apply the specific motion patterns—or "prototypes"—that define a dance style.

Methodology

The proposed framework addresses style-concentrated generation through two primary components: a Style Embedding Learner and a Music-to-Dance Generator.

The proposed framework architecture, illustrating the Style Embedding Learner and the Music-to-Dance Generator.

Style Embedding Learner

Inspired by Vector Quantized Variational Autoencoders (VQ-VAE), the model learns a Prototype Codebook. This codebook acts as a library of basic, abstract motion patterns shared across different styles. Rather than simply selecting the single closest move, the system employs a Multi-Head Self-Attention mechanism. This allows a specific style to be represented as a weighted linear combination of prototypes, creating a "style embedding" that is both descriptive and mathematically flexible. To ensure distinct boundaries between styles, the authors apply a Contrastive Loss function, forcing embeddings for different styles (e.g., Popping vs. Locking) to cluster separately in the latent space.

Music-to-Dance Generator

The generation process begins with a Transformer-based music encoder that captures long-range temporal patterns in the audio. This music representation is combined with the learned style embedding and an initial pose provided by a separate Initial Pose VAE. These inputs are fed into an LSTM-based generator that produces the final dance sequence frame-by-frame, ensuring that the resulting motion is both rhythmically aligned with the music and stylistically consistent.

Datasets

The study utilizes two primary datasets to train and validate the model:

  • AIST++: A standard benchmark for music-to-dance generation.
  • I-Dance: A new, large-scale dataset introduced by the authors to support multi-style research. It contains 1.48 million frames (at 30fps) across three distinct categories: Popping, Anime, and Locking. This dataset is significantly more style-concentrated than previous benchmarks, providing the necessary variety for robust style learning.

Key Results and Insights

The framework was evaluated against several state-of-the-art baselines, including Dancing2Music and DanceRevo.

PCA visualization of style embeddings: (a) without contrastive loss, showing overlapping styles; (b) with contrastive loss, showing clear separation between Popping, Locking, and Anime dance.

Style Separation and Consistency

A critical insight of this work is the importance of the contrastive loss in maintaining style purity. As shown in the PCA visualizations, without this loss, style embeddings tend to overlap, leading to ambiguous choreography. With contrastive loss, the model generates distinct, high-fidelity moves for each category even when the input music remains identical.

Quantitative and Qualitative Performance

  • Beat Alignment: The model achieved a top "Beat Align Score" of 0.245 on the AIST++ dataset, demonstrating that style control does not degrade rhythmic accuracy.
  • Diversity: Distance metrics confirmed that the generated dances were more diverse and less repetitive than those produced by baseline models.
  • User Preference: In a study with 18 participants, the system's output was preferred over baselines in 55% to 94% of cases, highlighting its superior aesthetic and stylistic quality.

Conclusion

By treating dance style as a "recipe" of motion prototypes, this framework moves beyond simple clip retrieval toward true stylized generation. While the current model masters "1st-order knowledge" (identifying which moves belong to which style), the authors note that future work will focus on "high-order knowledge"—the complex logic of how dancers transition between specific prototypes. Expanding the system to encompass a broader array of global dance styles remains a primary goal for scaling this technology.

Summary updated

BibTeX
@inproceedings{zhang2023mining,
  author = {Xinjian Zhang and Su Yang and Yi Xu and Weishan Zhang and Longwen Gao},
  title = {{Mining and Applying Composition Knowledge of Dance Moves for Style-Concentrated Dance Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i4.25673},
}
Figure