Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control

Zhe Li, Cheng Chi, Yangyang Wei, Boan Zhu, Tao Huang, Zhenguo Sun, Yibo Peng, Pengwei Wang, Zhongyuan Wang, Fangzhou Liu, Chang Xu, Shanghang ZhangarXiv 2025 · DOI · arXiv

Abstract

Humanoid robots often struggle with expressive, rhythmically aligned movements like dancing or gesturing during speech. Traditional "cascaded" pipelines—converting audio to human poses, retargeting to robot skeletons, and then tracking—are fragile, high-latency, and often lose the stylistic "vibe" of the input. We introduce RoboPerform, a framework that bypasses these intermediate steps to generate expressive humanoid locomotion directly from audio. By employing a novel Delta Mixture of Experts ($Δ$MoE) teacher policy and a diffusion-based student policy, RoboPerform achieves low-latency, physically plausible motion that maintains rhythmic and semantic alignment.

Introduction: The Challenge of Robotic "Soul"

While humanoid robots have made significant strides in functional tasks like walking and navigation, they often lack the expressive nuances of human movement. Current systems for expressive motion typically rely on a three-step process: pose estimation from audio, retargeting to the robot's kinematics, and controller tracking. This chain is prone to error accumulation and high latency, which prevents real-time interaction. RoboPerform addresses this by asking: Can we skip the middleman and map audio directly to robot actuation?

Methodology: The Teacher-Student Framework

RoboPerform is built on the principle that Motion = Content + Style. The framework utilizes a teacher-student architecture to learn and distill complex expressive behaviors.

The Teacher Policy ($Δ$MoE)

To handle the complexity of expressive motion, the researchers developed the Delta Mixture of Experts ($Δ$MoE). Unlike standard MoE architectures where experts might have overlapping responsibilities, $Δ$MoE uses a nested hierarchy. Each expert specializes in the "marginal gain" of adding specific information—such as basic physics, proprioception, or reference motion. This incremental training ensures that each part of the network contributes uniquely, enhancing robustness and efficiency.

Audio-Motion Alignment

A critical component is the alignment of audio features with physical motion. Using an adaptor trained with InfoNCE loss, the system maps raw audio into a latent space where rhythmic elements (like a drumbeat) correspond directly to physical "beats" in the motion space.

RoboPerform System Architecture: The framework consists of a Teacher Policy using Delta Mixture of Experts, an Audio-Motion Alignment module, and a Diffusion-based Student Policy.

The Student Policy (Diffusion)

The knowledge from the $Δ$MoE teacher is distilled into a diffusion-based student policy. This policy treats a text prompt (e.g., "A person is dancing") as the core Content and the aligned audio as the Style modulation. The use of a diffusion model allows the robot to capture the multi-modal nature of "freestyle" movement, where multiple valid motions can exist for a single audio input.

Experiments and Results

The model was trained on the FineDance (7.7 hours of music-dance pairs) and BEAT2 (76 hours of co-speech gestures) datasets. Evaluation was conducted in IsaacGym and MuJoCo simulations, as well as on a physical Unitree G1 humanoid.

Performance Metrics

RoboPerform achieved a 99% success rate on the BEAT2 speech task and 93% on the FineDance task in simulation. Crucially, the system demonstrated superior rhythmic alignment compared to traditional pose-driven baselines, as measured by the Beat Alignment Score (BAS).

Qualitative results of RoboPerform: The robot demonstrates expressive "freestyle" dancing and co-speech gesturing (talking) while maintaining physical balance.

Real-World Deployment

The student policy achieves an inference latency of just 5.3ms (using a 2-step DDIM sampling process), enabling real-time performance on physical hardware. The Unitree G1 successfully performed expressive motions in the real world, proving the framework's effectiveness in handling noisy physical dynamics.

Discussion and Future Work

RoboPerform represents a significant step toward more "human-like" robotic interaction. By eliminating the need for explicit pose reconstruction, it avoids the "uncanny valley" caused by kinematic mismatches. However, limitations remain. The robot's gestures are primarily reactive to prosody rather than deep semantic meaning, and physical hardware constraints still limit the range of "extreme" motions like breakdancing. Future work will focus on deeper semantic understanding and expanding the diversity of cultural dance styles in the training data.

Summary updated

Abstract

Humans intuitively move to sound, but current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands. Generating motion from audio and then retargeting it to robots relies on explicit motion reconstruction, leading to cascaded errors, high latency, and disjointed acoustic-actuation mapping. We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio. Guided by the core principle of "motion = content + style", the framework treats audio as implicit style signals and eliminates the need for explicit motion reconstruction. RoboPerform integrates a ResMoE teacher policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection. This retargeting-free design ensures low latency and high fidelity. Experimental validation shows that RoboPerform achieves promising results in physical plausibility and audio alignment, successfully transforming robots into responsive performers capable of reacting to audio.
BibTeX
@misc{li2025do,
  author = {Zhe Li and Cheng Chi and Yangyang Wei and Boan Zhu and Tao Huang and Zhenguo Sun and Yibo Peng and Pengwei Wang and Zhongyuan Wang and Fangzhou Liu and Chang Xu and Shanghang Zhang},
  title = {{Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control}},
  year = {2025},
  eprint = {2512.23650},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.23650},
}
Figure