POPDG: Popular 3D Dance Generation with PopDanceSet

Zhenye Luo, Min Ren, Xuecai Hu, Yongzhen Huang, Li Yao

TL;DR

The authors introduce PopDanceSet, the first aesthetics-driven 3D dance dataset tailored to modern youth preferences, alongside POPDG, a state-of-the-art diffusion-based generation model. By incorporating a novel Space Augmentation Algorithm and an improved diffusion framework (iDDPM), POPDG successfully balances dance diversity with physical realism, particularly in the upper body.

What Problem Does This Paper Address?

Generating realistic, expressive 3D human dance animation from music is a challenging cross-modal task. Current approaches face two primary bottlenecks:

  1. Dataset Limitations: Existing benchmarks like AIST++ are limited in style (mostly restricted to 10 subcategories of street dance) and lack the aesthetic variety seen in popular culture.
  2. Physical Realism vs. Diversity: While diffusion models have improved generation diversity, they often suffer from physical inconsistencies. Specifically, joint errors tend to accumulate the further a joint is from the body's root (the hip), leading to unnatural upper-body swaying or "floaty" movements. Furthermore, existing evaluation metrics like Physical Foot Contact (PFC) only assess lower-body realism, ignoring upper-body plausibility.

Methods

The proposed model, POPDG (Popular 3D Dance Generation), builds upon the improved-Denoising Diffusion Probabilistic Model (iDDPM) framework. Unlike standard DDPMs that only predict the noise mean, iDDPM also accounts for variance, which the authors show is key to balancing generation quality and diversity.

Figure 2. POPDG Pipeline Overview. POPDG, utilizing the iDDPM framework, learns to denoise dance sequences from time t = T to t = 0. The audio feature sequence serves as the input to the Music Encoder Block, while the noisy sequence is input to the Dance Decoder Block, with the output being the generated dance sequence. And N refers to the stack number. Beginning with a noisy sequence zT ∼N(0, I), POPDG generates the estimated frame of the dance sequence. It then progressively noises the sequence back to ˆzT −1, repeating the process until t = 0.

The architecture features two symmetrical blocks: a music encoder and a dance decoder. The core innovations within this pipeline include:

  • Dance Spatial Attention (DS-Attention) & Space Augmentation Algorithm: To combat upper-body joint drift, the authors introduce a spatial attention mechanism that explicitly weights the physical connections between joints based on their hierarchical distance from the hip. It dynamically passes structural constraints from parent joints down to downstream joints.
  • Alignment Module: A streamlined module that uses 1D convolutions and a Multi-Layer Perceptron (MLP) with DenseFiLM to fuse spatiotemporal features of both music and dance, ensuring tight rhythmic synchronization.
  • Physical Body Contact (PBC) Score: The authors extend the traditional foot-focused PFC metric into a full-body evaluation metric by incorporating acceleration and velocity constraints for the neck, chest, and hands.

Datasets

The paper evaluates its approach on two datasets:

  • AIST++: The standard baseline containing 10 street dance genres across 30 subjects (18,694 seconds).
  • PopDanceSet (New): A newly curated dataset harvested from the popular video platform Bilibili. The authors designed a statistical "popularity function" using multiple linear regression and t-tests to filter videos that align with modern youth aesthetics. It spans 19 genres (including CPOP, KPOP, and house dance), features 132 subjects, and totals 12,819 seconds of data. Crucially, it includes corresponding music lyrics.

Key Insights and Ideas

  • The "Whip Effect" in Pose Estimation: The authors quantitatively demonstrate that joint errors scale with their distance from the root joint. While the root joint error sits around 4%, upper-body extremities like the neck and ribs experience errors up to 6.5%.
Figure 3. Analysis of Joint Error Distribution in SMPL Hu- man Body Model. (a) SMPL Joint Labeling: Marks human body joints from the hip (level 0 joint) outward, color-coded by different levels. (b) Joint Error Proportions: Shows that upper body joints experience increasing error the further they are from the hip. (c) Upper Body Joint Error Levels: Displays average errors across up- per body joint levels.
  • Injecting Kinematic Structure into Attention: Standard multi-head attention calculates weights purely on feature similarity. By forcing the attention map to respect the human skeletal hierarchy via the Space Augmentation Algorithm, the model preserves physical realism even when generating highly diverse and complex movements.
Figure 4. The Overview of Dance Spatial Attention. The key distinction between dance spatial attention and standard multi- head attention is the incorporation of the Space Augmentation Al- gorithm when calculating the Attention Map between Query and Key. This algorithm is tailored to emphasize the upper body joints in relation to the hip, enhancing their spatial inter connectivity.

Key Results

POPDG achieves new state-of-the-art (SOTA) results across both datasets:

  • On PopDanceSet: POPDG significantly outperformed the competitive EDGE baseline, lowering the PFC score from 5.9701 to 4.3697 (lower is better, closer to ground truth) and improving kinematic diversity ($Div_k$) from 3.6065 to 4.8641.
  • On AIST++: It achieved a SOTA PFC score of 0.8014 and a Beat Alignment Score of 0.469, outperforming FACT, Bailando, and EDGE.
  • User Study: In a blind evaluation with 20 participants comparing model outputs, 70% of users preferred the dances generated from PopDanceSet over those from AIST++, validating that the new dataset successfully captures more engaging, contemporary choreography.

Limitations and Future Work

Despite the impressive generation quality, the model's end-to-end training footprint is heavy, requiring approximately 100 hours on two A800 GPUs for AIST++ and 66 hours for PopDanceSet. The authors note that future work will focus on adapting these spatial attention mechanisms into more lightweight architectures to reduce computational overhead. Additionally, they plan to explore deeper cross-modal relationships between dance movements and music lyrics.

Summary updated

Abstract

Generating dances that are both lifelike and well-aligned with music continues to be a challenging task in the cross-modal domain. This paper introduces PopDanceSet, the first dataset tailored to the preferences of young audiences, enabling the generation of aesthetically oriented dances. And it surpasses the AIST++ dataset in music genre diversity and the intricacy and depth of dance movements. Moreover, the proposed POPDG model within the iDDPM framework enhances dance diversity and, through the Space Augmentation Algorithm, strengthens spatial physical connections between human body joints, ensuring that increased diversity does not compromise generation quality. A streamlined Alignment Module is also designed to improve the temporal alignment between dance and music. Extensive experiments show that POPDG achieves SOTA results on two datasets. Furthermore, the paper also expands on current evaluation metrics. The dataset and code are available at https://github.com/Luke-Luo1/POPDG.
BibTeX
@inproceedings{luo2024popdg,
  author = {Zhenye Luo and Min Ren and Xuecai Hu and Yongzhen Huang and Li Yao},
  title = {{POPDG: Popular 3D Dance Generation with PopDanceSet}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure