Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation

Guangtao Lyu, Chenghao Xu, Qi Liu, Jiexi Yan, Muli Yang, Fen Fang, Cheng DengarXiv 2025 · arXiv · OpenReview

Abstract

Music to 3D dance generation aims to synthesize realistic and rhythmically synchronized human dance from music. While existing methods often rely on additional genre labels to further improve dance generation, such labels are typically noisy, coarse, unavailable, or insufficient to capture the diversity of real-world music, which can result in rhythm misalignment or stylistic drift. In contrast, we observe that tempo, a core property reflecting musical rhythm and pace, remains relatively consistent across datasets and genres, typically ranging from 60 to 200 BPM. Based on this finding, we propose TempoMoE, a hierarchical tempo-aware Mixture-of-Experts module that enhances the diffusion model and its rhythm perception. TempoMoE organizes motion experts into tempo-structured groups for different tempo ranges, with multi-scale beat experts capturing fine- and long-range rhythmic dynamics. A Hierarchical Rhythm-Adaptive Routing dynamically selects and fuses experts from music features, enabling flexible, rhythm-aligned generation without manual genre labels. Extensive experiments demonstrate that TempoMoE achieves state-of-the-art results in dance quality and rhythm alignment.

Introduction: The Limitations of Genre-Based Guidance

Generating realistic 3D dance from music requires a delicate balance between physical realism and rhythmic synchronization. Most existing models rely on genre labels (e.g., "Pop" or "Breakdance") to guide the generation process. However, this approach suffers from several critical flaws identified by the authors:

  1. Coarse Granularity: A single genre tag cannot capture the vast differences in motion required for songs within that genre at different tempos. For instance, a "Pop" song at 60 BPM requires slow, expansive movements, whereas the same genre at 180 BPM demands rapid, localized isolations.
  2. Unreliable Supervision: Many real-world datasets have noisy, inconsistent, or entirely missing genre labels, limiting the scalability of supervised models.
  3. Limited Adaptability: Fixed genre tokens struggle to accommodate the evolving nature of musical hybrids and subgenres.

The authors propose that tempo (Beats Per Minute) is a more stable and universal cue. Tempo dictates the "time budget" for each move, acting as a physical constraint that remains consistent across different musical styles.

Methodology: The TempoMoE Architecture

The core innovation of this work is TempoMoE, a module integrated into a Denoising Diffusion Transformer (DiT). TempoMoE utilizes a hierarchical structure to process rhythmic information at multiple scales.

The TempoMoE architecture, featuring hierarchical routing and specialized experts for different tempo ranges and beat resolutions.

Hierarchical Expert Organization

  • Tempo-Structured Expert Groups: The 60–200 BPM range is discretized into eight overlapping bands. Each band is assigned a dedicated group of experts with temporal receptive fields (convolutional kernel sizes) mathematically tuned to that specific tempo.
  • Multi-Scale Beat Experts: Within each tempo group, three specialized experts handle different rhythmic resolutions: quarter-beat (fine accents), half-beat, and whole-beat (long-range phrasing).

Hierarchical Rhythm-Adaptive Routing

The model employs a two-stage gating mechanism to dynamically select experts:

  1. Hard Tempo Routing: A "TempoGateNet" selects the top-2 most relevant tempo groups based on global music features.
  2. Soft Beat Routing: A "BeatGateNet" assigns soft weights to fuse the three beat-scale experts within the selected groups, allowing for a fluid blend of micro-movements and macro-gestures.

Tempo as a Physical Prior

A key insight of this research is that tempo defines the physics of motion. High BPM forces "compact" motion because the dancer lacks the time for full-body transitions between beats. Conversely, lower BPM allows for more expansive, large-scale movements.

Comparison of dance motion at different tempos. High BPM leads to rapid small-scale motion, while low BPM allows for large-scale movements.

By routing through tempo-specific experts, the model learns to adapt its pacing independently of the specific dance style, leading to significantly improved beat alignment.

Experimental Results

TempoMoE was evaluated on three major benchmarks: AIST++, FineDance, and PopDanceSet. The latter is particularly challenging as it lacks genre labels, making it an ideal test for a tempo-based approach.

Performance and Efficiency

  • State-of-the-Art Accuracy: TempoMoE outperformed previous leaders like EDGE and Lodge across all datasets, particularly in the Beat Alignment Score (BAS).
  • Qualitative Quality: The model effectively avoids "motion freezing" and "genre drift," maintaining stylistic consistency throughout the sequence.
  • Inference Speed: By requiring only 10 denoising steps, TempoMoE achieves a 2x–5x speedup over baseline models, generating 1024 frames of motion in approximately 0.6 seconds on an NVIDIA A6000.

Conclusion

TempoMoE shifts the paradigm of music-to-dance generation by replacing noisy genre labels with tempo as the primary rhythmic anchor. By utilizing a hierarchical MoE architecture that specializes in specific tempo ranges and beat resolutions, the model achieves superior rhythm alignment and motion quality with significant computational efficiency.

Summary updated

Abstract

Music to 3D dance generation aims to synthesize realistic and rhythmically synchronized human dance from music. While existing methods often rely on additional genre labels to further improve dance generation, such labels are typically noisy, coarse, unavailable, or insufficient to capture the diversity of real-world music, which can result in rhythm misalignment or stylistic drift. In contrast, we observe that tempo, a core property reflecting musical rhythm and pace, remains relatively consistent across datasets and genres, typically ranging from 60 to 200 BPM. Based on this finding, we propose TempoMoE, a hierarchical tempo-aware Mixture-of-Experts module that enhances the diffusion model and its rhythm perception. TempoMoE organizes motion experts into tempo-structured groups for different tempo ranges, with multi-scale beat experts capturing fine- and long-range rhythmic dynamics. A Hierarchical Rhythm-Adaptive Routing dynamically selects and fuses experts from music features, enabling flexible, rhythm-aligned generation without manual genre labels. Extensive experiments demonstrate that TempoMoE achieves state-of-the-art results in dance quality and rhythm alignment.
BibTeX
@misc{lyu2025tempo,
  author = {Guangtao Lyu and Chenghao Xu and Qi Liu and Jiexi Yan and Muli Yang and Fen Fang and Cheng Deng},
  title = {{Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation}},
  year = {2025},
  eprint = {2512.18804},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18804},
}
Figure