Dancing to Music

Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz

Overview

Researchers have developed a novel "decomposition-to-composition" framework that generates realistic, style-consistent, and beat-matched dance sequences from raw music. By breaking dance down into basic "units" and learning to reassemble them based on musical features, the model moves beyond simple video retrieval to true cross-modal synthesis.

The Challenge of Choreography Generation

Generating dance from music is a classic "hard" problem in AI for three main reasons:

  1. Cross-Modal Synchronization: The movement must not only look realistic but also align precisely with the rhythm (beats) and "vibe" (style) of the audio.
  2. Multimodality: There isn't just one "right" dance for a song. A single pose can be followed by an infinite variety of movements, making deterministic models (like standard LSTMs) prone to "collapsing" into static or repetitive poses.
  3. Kinematic Complexity: Human movement has intricate spatio-temporal structures that are difficult to model over long durations without the motion becoming "floaty" or incoherent.

Existing methods often treated this as a retrieval task—searching a database for the best-fitting pre-recorded clip. This paper shifts the paradigm toward a generative approach, aiming to create entirely new choreography.

Methods: Decomposition and Composition

The authors propose a two-phase framework: Decomposition (learning how to move) and Composition (learning how to choreograph).

The proposed decomposition-to-composition framework for dance generation.

1. Decomposition (Analysis Phase)

The model first learns to represent dance as a series of "dance units"—short, temporally standardized snippets of motion.

  • Kinematic Beat Detector: To define "beats" in movement, the authors developed a detector that identifies sudden decelerations or changes in direction (offsets).
  • DU-VAE (Dance Unit VAE): This Variational Autoencoder disentangles a dance unit into two latent spaces: an initial pose space (where the movement starts) and a movement space (the actual motion). This disentanglement is crucial because it allows the model to "stitch" units together seamlessly by ensuring the end of one unit matches the start of the next.

2. Composition (Synthesis Phase)

Once the model understands basic movements, it needs to learn how to sequence them based on music.

  • MM-GAN (Music-to-Movement GAN): This generative adversarial network takes music style features and audio beats as input. It learns to map these musical cues to a sequence of movement codes.
  • Beat Warper: In the final step, the model doesn't just play the units back-to-back. It uses a "beat warper" to stretch or shrink the generated dance units so that their internal kinematic beats align perfectly with the detected musical beats.

Datasets and Key Insights

Because no large-scale paired music-dance dataset existed, the authors curated a massive new benchmark containing over 360,000 video clips totaling ~71 hours of footage across Ballet, Zumba, and Hip-Hop. They used OpenPose to extract 2D skeletal keypoints (14 joints) and MFCC (Mel-frequency cepstral coefficients) for audio features.

Key Insights: * Movement has "Beats" too: The most profound insight is the formalization of the "kinematic beat." By treating physical deceleration as the visual equivalent of an audio onset, the researchers created a mathematical bridge between two very different types of data. * Synthesis-by-Analysis: The paper argues that you cannot learn to choreograph until you learn the "vocabulary" of movement. By forcing the model to first master discrete dance units, they avoid the "drifting" effect common in end-to-end motion models. * Style vs. Rhythm: The framework cleverly separates style (handled by the GAN's latent space) from rhythm (handled by the explicit beat warping), allowing for dances that are both "on beat" and "in character."

Key Results

The model was evaluated against baselines like Aud-MoCoGAN and standard LSTMs using both quantitative metrics and a user study.

Qualitative comparison of generated dances against baseline models like LSTM and Aud-MoCoGAN.
  • Realism and Style: In a "blind" user study, 61.2% of participants preferred the proposed model's realism over baselines. For style consistency, 68.3% preferred it.
  • FID (Fréchet Inception Distance): The model achieved an FID of 12.8, significantly lower (better) than the LSTM baseline (73.8), indicating the generated dance distribution closely matches real human dancing.
  • Beat Matching: The model achieved a "Beat Coverage" of 39.4%, which is nearly identical to the 39.3% found in real human dance data, proving the effectiveness of the beat-warping mechanism.

Limitations and Future Work

While the results are impressive, the authors note a few hurdles:

  • Detector Sensitivity: The kinematic beat detector is an approximation; it can miss subtle movements that a human would perceive as a beat.
  • Data Noise: Since the data was scraped from the web, "noisy" pose estimations (due to occlusions or complex lighting) can occasionally lead to artifacts in the generated skeletons.
  • Expanding Genres: The current model is trained on solo performances. Future work could explore more complex social dynamics, such as partner dancing or group choreography.

Summary updated

Abstract

Dancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem. It requires significant efforts to measure the correlation between music and dance as one needs to simultaneously consider multiple aspects, such as style and beat of both music and dance. Additionally, dance is inherently multimodal and various following movements of a pose at any moment are equally likely. In this paper, we propose a synthesis-by-analysis learning framework to generate dance from music. In the top-down analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move. In the bottom-up synthesis phase, the model learns how to compose a dance by combining multiple basic dancing movements seamlessly according to input music. Experimental qualitative and quantitative results demonstrate that the proposed method can synthesize realistic, diverse, style-consistent, and beat-matching dances from music.
BibTeX
@inproceedings{lee2019dancing,
  author = {Hsin-Ying Lee and Xiaodong Yang and Ming-Yu Liu and Ting-Chun Wang and Yu-Ding Lu and Ming-Hsuan Yang and Jan Kautz},
  title = {{Dancing to Music}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure