Abstract
Humanoid robots often struggle with expressive, rhythmically aligned movements like dancing or gesturing during speech. Traditional "cascaded" pipelines—converting audio to human poses, retargeting to robot skeletons, and then tracking—are fragile, high-latency, and often lose the stylistic "vibe" of the input. We introduce RoboPerform, a framework that bypasses these intermediate steps to generate expressive humanoid locomotion directly from audio. By employing a novel Delta Mixture of Experts ($Δ$MoE) teacher policy and a diffusion-based student policy, RoboPerform achieves low-latency, physically plausible motion that maintains rhythmic and semantic alignment.
Introduction: The Challenge of Robotic "Soul"
While humanoid robots have made significant strides in functional tasks like walking and navigation, they often lack the expressive nuances of human movement. Current systems for expressive motion typically rely on a three-step process: pose estimation from audio, retargeting to the robot's kinematics, and controller tracking. This chain is prone to error accumulation and high latency, which prevents real-time interaction. RoboPerform addresses this by asking: Can we skip the middleman and map audio directly to robot actuation?
Methodology: The Teacher-Student Framework
RoboPerform is built on the principle that Motion = Content + Style. The framework utilizes a teacher-student architecture to learn and distill complex expressive behaviors.
The Teacher Policy ($Δ$MoE)
To handle the complexity of expressive motion, the researchers developed the Delta Mixture of Experts ($Δ$MoE). Unlike standard MoE architectures where experts might have overlapping responsibilities, $Δ$MoE uses a nested hierarchy. Each expert specializes in the "marginal gain" of adding specific information—such as basic physics, proprioception, or reference motion. This incremental training ensures that each part of the network contributes uniquely, enhancing robustness and efficiency.
Audio-Motion Alignment
A critical component is the alignment of audio features with physical motion. Using an adaptor trained with InfoNCE loss, the system maps raw audio into a latent space where rhythmic elements (like a drumbeat) correspond directly to physical "beats" in the motion space.
The Student Policy (Diffusion)
The knowledge from the $Δ$MoE teacher is distilled into a diffusion-based student policy. This policy treats a text prompt (e.g., "A person is dancing") as the core Content and the aligned audio as the Style modulation. The use of a diffusion model allows the robot to capture the multi-modal nature of "freestyle" movement, where multiple valid motions can exist for a single audio input.
Experiments and Results
The model was trained on the FineDance (7.7 hours of music-dance pairs) and BEAT2 (76 hours of co-speech gestures) datasets. Evaluation was conducted in IsaacGym and MuJoCo simulations, as well as on a physical Unitree G1 humanoid.
Performance Metrics
RoboPerform achieved a 99% success rate on the BEAT2 speech task and 93% on the FineDance task in simulation. Crucially, the system demonstrated superior rhythmic alignment compared to traditional pose-driven baselines, as measured by the Beat Alignment Score (BAS).
Real-World Deployment
The student policy achieves an inference latency of just 5.3ms (using a 2-step DDIM sampling process), enabling real-time performance on physical hardware. The Unitree G1 successfully performed expressive motions in the real world, proving the framework's effectiveness in handling noisy physical dynamics.
Discussion and Future Work
RoboPerform represents a significant step toward more "human-like" robotic interaction. By eliminating the need for explicit pose reconstruction, it avoids the "uncanny valley" caused by kinematic mismatches. However, limitations remain. The robot's gestures are primarily reactive to prosody rather than deep semantic meaning, and physical hardware constraints still limit the range of "extreme" motions like breakdancing. Future work will focus on deeper semantic understanding and expanding the diversity of cultural dance styles in the training data.
Summary updated