Dancing to Music

Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz

Abstract

Dancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem. It requires significant efforts to measure the correlation between music and dance as one needs to simultaneously consider multiple aspects, such as style and beat of both music and dance. Additionally, dance is inherently multimodal and various following movements of a pose at any moment are equally likely. In this paper, we propose a synthesis-by-analysis learning framework to generate dance from music. In the analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move. In the synthesis phase, the model learns how to compose a dance by organizing multiple basic dancing movements seamlessly according to the input music. Experimental qualitative and quantitative results demonstrate that the proposed method can synthesize realistic, diverse, style-consistent, and beat-matching dances from music.

Introduction

Generating dance from music is a complex cross-modal task. While humans synchronize movement to rhythm effortlessly, machines face significant hurdles. First, rhythmic synchronization requires aligning movement with both the genre's style and specific musical beats. Second, dance is multimodal; given a pose and a song, many subsequent moves are valid, making deterministic models prone to producing "average," blurry results. Finally, the high degrees of freedom in human motion and the long-term temporal structures of choreography are difficult to model over extended sequences.

Previous approaches often relied on retrieval-based methods, stitching together pre-recorded clips. This work moves toward pure generation, proposing a framework that creates entirely new choreography from scratch.

Methods: Synthesis-by-Analysis

The proposed framework operates in two distinct phases: decomposition and composition.

The proposed synthesis-by-analysis framework, consisting of decomposition (learning dance units), composition (music-driven organization), and testing phases.

1. The Analysis Phase (Decomposition)

To learn "how to move," the model decomposes complex dances into "atomic" dance units. A Kinematic Beat Detector identifies moments where motion changes direction or slows significantly. A Dance Unit Variational Autoencoder (DU-VAE) then disentangles these units into two latent spaces: the initial pose and the movement code. This separation allows the model to "stitch" units together seamlessly by ensuring the end of one unit matches the start of the next.

2. The Synthesis Phase (Composition)

The Music-to-Movement GAN (MM-GAN) learns "how to compose." It extracts style features and beats from raw audio, then organizes movement codes from the DU-VAE into a sequence matching the music's vibe. A Beat Warper finally applies temporal warping to ensure kinematic beats align precisely with musical beats.

Key Insights

A core innovation is the use of kinematic beats as a bridge between modalities. By normalizing dance into units representing movement between beats, the model treats choreography as a series of standardized building blocks. Furthermore, by disentangling style from rhythm, the model can generate diverse movements appropriate for specific genres like Ballet, Zumba, or Hip-Hop.

Experimental Results

The model was evaluated on a newly curated dataset of over 360,000 video clips (71 hours) across three genres: Ballet, Zumba, and Hip-Hop.

Qualitative comparison: (a) Our model produces more realistic poses than LSTM and Aud-MoCoGAN baselines. (b) Diverse dance sequences generated from different musical inputs.

Quantitative results highlight the framework's superiority:

  • Realism: The model achieved a Fréchet Inception Distance (FID) of 12.8, significantly outperforming the LSTM baseline (73.8) and approaching real dance (5.9).
  • User Preference: In blind tests, human evaluators preferred the proposed model's realism and style consistency over 60% of the time compared to other generative baselines.
  • Beat Matching: The "Beat Hit Rate" was found to be comparable to real human dancers, validating the effectiveness of the Beat Warper.

Conclusion

By breaking dance down into fundamental units and reassembling them through a style-aware GAN, this framework achieves a new level of realism in music-to-dance generation. Future work will aim to address data noise from automatic pose extraction and expand the model to handle more complex social and partner-based dancing styles.

Reference: Lee, H. Y., et al. (2019). Dancing to Music. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).

Summary updated

Abstract

Dancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem. It requires significant efforts to measure the correlation between music and dance as one needs to simultaneously consider multiple aspects, such as style and beat of both music and dance. Additionally, dance is inherently multimodal and various following movements of a pose at any moment are equally likely. In this paper, we propose a synthesis-by-analysis learning framework to generate dance from music. In the analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move. In the synthesis phase, the model learns how to compose a dance by organizing multiple basic dancing movements seamlessly according to the input music. Experimental qualitative and quantitative results demonstrate that the proposed method can synthesize realistic, diverse,style-consistent, and beat-matching dances from music.
BibTeX
@inproceedings{lee2019dancing,
  author = {Hsin-Ying Lee and Xiaodong Yang and Ming-Yu Liu and Ting-Chun Wang and Yu-Ding Lu and Ming-Hsuan Yang and Jan Kautz},
  title = {{Dancing to Music}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure