TL;DR
OmniHumanoid is a novel video generation framework that scales up data production for embodied AI by transferring human or robot motions to new robot embodiments. By decoupling transferable motion dynamics from robot-specific appearances, it allows quick adaptation to unseen robots using only unpaired videos and achieves real-time, streaming 720p video generation via a causal distillation pipeline.
What Problem Does This Paper Address?
Training humanoid robots requires vast amounts of video data showing how robots interact with the physical world. However, physically collecting high-quality demonstrations for every new robot model is prohibitively expensive and slow.
An appealing alternative is cross-embodiment video generation—taking an existing video (e.g., a human performing a task) and "robotizing" it so it looks like a specific target robot executing the exact same action.
Current methods struggle with this because they entangle motion with the robot's unique geometry and kinematics. Furthermore, most existing frameworks require expensive paired data (the exact same motion sequence performed by both the source and target embodiment) for every new robot model, severely limiting scalability. They are also typically too slow for real-time or interactive data generation.
Methods
OmniHumanoid operates under the TAPE principle: Transferable motion, Paired-free adaptation, embodiment Preservation, and generation Efficiency. Built on a Diffusion Transformer (DiT) backbone, the framework splits video generation into two independent pathways:
- Shared Motion Transfer Model: This module runs on a conditioning branch, learning embodiment-invariant features like temporal dynamics, object interactions, and scene evolution from a multi-embodiment dataset.
- Embodiment Video LoRAs: These are lightweight Low-Rank Adaption modules attached strictly to the denoising branch. Each LoRA is dedicated to a specific robot, capturing its unique geometry, textures, and structural proportions.
To keep these components from interfering with one another, the authors introduce a Unidirectional Information Flow design using an asymmetric attention mask. The denoising branch can read motion cues from the conditioning branch, but the motion branch is completely blinded to the embodiment-specific details.
Streaming Video-to-Video Distillation
To make the model practical for large-scale data generation, the authors distilled the computationally heavy bidirectional teacher model into a causal streaming student model. By organizing tokens into an interleaved "super-chunk" layout, the model generates video autoregressively. This distillation compresses the generation process from 50 denoising steps down to just 4 steps.
Datasets
The authors introduced a massive new synthetic dataset and paired it with real-world testing:
- Human-Humanoid 4D Dataset: Built using Unity HDRP and the Humoto motion library, this dataset features over 700 humanoid motion sequences retargeted across 10 distinct assets (5 robots, 5 human avatars) in 100 diverse scenes. It totals 167 hours and 14.4 million frames of perfectly motion-aligned paired videos.
- Evaluation Split: To rigorously test paired-free adaptation, one robot embodiment (Unitree-G1) was completely held out from training and used exclusively for testing.
- Real-world Benchmark: 50 in-the-wild videos of human daily activities and online robot demonstrations were collected to evaluate performance under realistic, unconstrained visual conditions.
Key Insights and Ideas
- Decoupling Prevents Interference: Directly mixing source-motion conditioning with target-embodiment rendering causes severe visual artifacts. Forcing a unidirectional information flow prevents embodiment priors from polluting the shared motion model.
- Zero-Shot-Like Scaling via Unpaired Data: Because the shared motion model handles "how things move," integrating an entirely new robot doesn't require paired motion data. Engineers only need to record a few dozen unpaired videos of the new robot to train a lightweight appearance adapter.
- Autoregressive Video Generation: Treating video tokens like block-wise causal chunks allows video diffusion models to function similarly to large language models, enabling infinite-horizon streaming generation.
Key Results
OmniHumanoid significantly outperforms state-of-the-art commercial and open-source video-to-video baselines (including Kling, Runway Gen-4, and X-Humanoid):
- Higher Fidelity: On the synthetic benchmark for the held-out Unitree-G1 robot, OmniHumanoid achieved a PSNR of 25.47 and SSIM of 0.9039, comfortably beating X-Humanoid (PSNR 23.03, SSIM 0.8891).
- Real-Time Speedups: The streaming distillation pipeline boosted generation speeds from a sluggish 0.10 FPS (teacher model) to 4.96 FPS (student model) at 720p resolution on a single NVIDIA H200 GPU.
- Human Preference: In a user study evaluating motion fidelity and embodiment similarity, OmniHumanoid won 72.7% and 65.7% of user preference votes, respectively, compared to less than 17% for commercial models like Kling O3.
Limitations and Future Work
The primary limitation lies in the speed-quality trade-off of the distillation process. While the 4-step streaming student model is fast enough for real-time applications, it suffers a noticeable drop in fine-grained details, temporal smoothness, and complex motion fidelity when compared directly to the high-quality outputs of the 50-step teacher model.
The authors note that current distillation frameworks struggle to perfectly approximate complex data distributions in so few steps, marking advanced knowledge distillation strategies as a key focus for future work.
Summary updated