Music2Dance: Music-driven Dance Generation using WaveNet

Wenlin Zhuang, Congyi Wang, Siyu Xia, Jinxiang Chai, Yangang WangarXiv 2020 · OpenReview

TL;DR

This paper introduces DanceNet, a novel autoregressive model that generates realistic and diverse 3D dance motions synchronized with music. By leveraging high-level music features (style, rhythm, melody) and a custom dataset of professional dance performances, DanceNet achieves state-of-the-art results in music-driven dance generation.

What Problem Does This Paper Address?

Generating realistic and engaging human dance motions that are perfectly synchronized with music is a significant challenge. Previous approaches often struggled with producing complex, diverse, and truly music-consistent movements. The core problem lies in capturing the intricate relationship between musical elements like style, rhythm, and melody, and translating them into fluid, expressive human motion. Existing datasets often lack the necessary synchronized music-dance pairs or the quality and diversity required for training such sophisticated models.

Methods

The authors propose DanceNet, an autoregressive generative model. Key to its design are:

  • High-Level Music Features: Instead of low-level audio features, DanceNet uses onset, beat (for rhythm), and chroma (for melody) as control signals. A musical style classifier further categorizes music into "smoothing" or "fast-rhythm" to guide dance type.
  • Autoregressive Generation with Dilated Convolutions: DanceNet employs a WaveNet-inspired architecture with dilated convolutions. This allows for a large receptive field to capture long-term dependencies in dance motion without increasing model depth, making it more robust than traditional RNNs.
  • Gated Activation Units: Similar to WaveNet and PixelCNN, gated activation units are incorporated to enhance the model's ability to capture complex and diverse dance patterns.
  • Contact-Aware Motion Representation: The model explicitly considers foot contact and uses end-effector positions (like toes and head) in its motion representation to improve realism and reduce error accumulation.
  • Gaussian Mixture Model (GMM) Loss: To better model the probabilistic distribution of dance motions and encourage diversity, a GMM loss function is used instead of the standard Mean Squared Error (MSE).
  • Post-processing for Foot Sliding: A dedicated "Foot Constraint Model" is introduced to address foot sliding, a common artifact in motion generation, by learning to correct lower body joint rotations.

Datasets

The paper introduces a newly created high-quality music-dance pair dataset. This dataset was captured using professional dancers and includes:

  • Modern dance (approx. 26.15 minutes)
  • Curtilage dance (approx. 31.72 minutes)
  • Motions are accurately aligned with their corresponding music.
  • The motion data includes 55 joints, notably including finger motion, which contributes to higher dance artistry.

Existing datasets like CMU Motion Capture, SFU Motion Capture, and Mixamo are mentioned as lacking dance-specific motion or synchronized music.

Key Insights and Ideas

  • High-level music features are crucial: Using onset, beat, and chroma provides a more meaningful representation for dance generation than raw audio features like Mel spectrum, leading to more diverse and music-consistent results.
  • Dilated convolutions are key for dance: The ability of dilated convolutions to expand the receptive field without increasing depth is vital for modeling the long-term temporal dependencies inherent in dance.
  • GMM loss promotes diversity: Moving beyond MSE loss to a GMM loss allows the model to represent a distribution of possible motions, enabling more varied and less repetitive dance generation.
  • End-effector positions are important for realism: Including end-effector positions in the motion representation helps mitigate error accumulation and leads to more accurate and diverse dance.
  • A dedicated dataset is essential: The creation of a high-quality, synchronized music-dance dataset was critical for training a model that could achieve state-of-the-art performance.

Key Results

  • Musical Style Classification: DanceNet achieved an accuracy of 92.1% in classifying musical styles, outperforming baseline Convolutional RNN and FC+Bi-LSTM models.
  • Realism and Style-Consistency (FID): DanceNet achieved the lowest FID scores (12.5 for modern dance, 8.7 for curtilage dance), indicating its generated motions are closest to real dances and more style-consistent.
  • Diversity: DanceNet demonstrated superior diversity compared to other methods, with scores of 52.5 (modern) and 46.9 (curtilage).
  • Rhythm-Consistency (Rhythm Hit Rate): The model achieved a rhythm hit rate of 58.7% (modern) and 69.2% (curtilage), outperforming most baselines.
  • User Study: A user study with 25 participants rated DanceNet's generated motions significantly higher than LSTM (GMM loss) for realism, diversity, and music-consistency, with mean scores of 8.452 (modern) and 8.196 (curtilage).
  • Post-processing: The proposed Foot Constraint Model significantly reduced Mean Squared Error (MSE) for foot sliding compared to a baseline LSTM approach.

Limitations and Future Work

  • Generated motions not as good as professional dancers: The authors acknowledge that the generated dance motions, while realistic and diverse, still fall short of the artistry of professional dancers.
  • Slight foot sliding: Despite the post-processing step, some slight foot sliding remains.

Future work will focus on improving the quality of generated dance motions to match professional dancers and further reducing foot sliding.

Summary updated

Abstract

Synthesize human motions from music, i.e., music to dance, is appealing and attracts lots of research interests in recent years. It is challenging due to not only the requirement of realistic and complex human motions for dance, but more importantly, the synthesized motions should be consistent with the style, rhythm and melody of the music. In this paper, we propose a novel autoregressive generative model, DanceNet, to take the style, rhythm and melody of music as the control signals to generate 3D dance motions with high realism and diversity. To boost the performance of our proposed model, we capture several synchronized music-dance pairs by professional dancers, and build a high-quality music-dance pair dataset. Experiments have demonstrated that the proposed method can achieve the state-of-the-art results.
BibTeX
@misc{zhuang2020music,
  author = {Wenlin Zhuang and Congyi Wang and Siyu Xia and Jinxiang Chai and Yangang Wang},
  title = {{Music2Dance: Music-driven Dance Generation using WaveNet}},
  year = {2020},
  eprint = {2002.03761},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.03761},
}
Figure