TL;DR
DiWALI is a novel framework that enables robots to learn complex manipulation skills from action-free human videos by inferring "latent actions" between frames. By using these latent actions to build a world model, the system can train and refine diffusion policies entirely in "imagination," significantly outperforming traditional methods that rely on scarce, action-labeled robot data.
What Problem Does This Paper Address?
The "robot-time bottleneck" is a major hurdle in scaling robot learning. Collecting high-quality robot data usually requires teleoperation, where one hour of data takes one hour of human and robot time. In contrast, humans can demonstrate tasks in seconds, and the internet is full of human videos.
However, these human videos lack action labels—the specific motor commands (joint torques or end-effector velocities) a robot needs. Furthermore, there is an embodiment gap: a human hand looks and moves differently than a robotic gripper. Most current world models (which act as internal simulators for robots) require action labels to learn the underlying physics. DiWALI aims to bridge this gap, allowing robots to learn from "play" data and demonstrations of humans without needing any robot-specific action signals.
Methods
The DiWALI (Diffusion Policy Learning in World Models via Latent Action Inference) framework operates in four distinct stages:
- Latent-Action World Model: Instead of conditioning a world model on robot commands, the authors use an inverse dynamics model to infer a "latent action" ($u_t$) from two consecutive video frames. This latent action captures the effect of a movement regardless of whether a human or a robot performed it. This allows a Recurrent State-Space Model (RSSM) to learn scene dynamics from both human and robot "play" data.
- Action Decoder: To make these latent actions executable, a small decoder is trained separately on a small amount of robot play data. It learns to map a latent action and the current state to a real robot command.
- Action-Free Policy Learning: A diffusion policy is trained to imitate human demonstrations. Because the policy operates in the latent action space, it can learn directly from human videos without needing to know the underlying motor commands.
- Policy Improvement in Imagination: The learned world model serves as a simulator. The policy is fine-tuned using Reinforcement Learning (PPO) within this "imagination" space, allowing the robot to improve its skills without further real-world interaction.
Datasets
- CALVIN: A standard benchmark for long-horizon robot manipulation.
- CALVIN-H: A new dataset introduced by the authors where the robot in CALVIN is replaced by a rendered human arm, and all action labels are removed to simulate human video data.
- Real-World Data: The authors collected 2 hours of robot play and 2 hours of human play (using a wrist-mounted camera to match the robot's perspective) in a tabletop environment.
Key Insights and Ideas
- Latent Actions as a Universal Language: By inferring actions from visual changes rather than motor commands, the model creates a shared representation for different embodiments. This effectively bypasses the need for manual retargeting of human motions to robot joints.
- The Importance of State-Conditioned Decoding: A key "aha moment" in the paper is that a latent action (e.g., "move right") requires different motor commands depending on the robot's current configuration. Conditioning the action decoder on the latent state was critical, nearly tripling success rates in ablations.
- Separate vs. Joint Training: The authors found that training the action decoder after the world model is frozen is superior to joint training. Joint training tends to "collapse" the latent space toward only what the robot can do, losing the rich diversity of movement found in human data.
- Human Play as a Simulator Booster: Including human play data doesn't just help the policy; it makes the world model a better simulator. This leads to more robust fine-tuning in imagination compared to using robot data alone.
Key Results
DiWALI demonstrates a significant leap in data efficiency:
- Outperforming Baselines: On the CALVIN benchmark, a DiWALI policy trained with zero robot demonstrations achieved a 59.6% success rate, outperforming a standard Diffusion Policy trained with five robot demonstrations (35.2%).
- Imagination Works: In real-world experiments, skills learned primarily from human videos improved from 53% to 70% success after fine-tuning in the world model's imagination, with no additional physical robot time required.
- Ablation Success: The choice to condition the decoder on the latent state and train it separately was the most impactful design decision, raising success from 18.7% to 61.6% in controlled tests.
Limitations and Future Work
While DiWALI reduces the need for action labels, it still requires some robot play data to ground the action decoder. The authors also note that the current model is trained on human data collected in the same workspace as the robot.
Future directions include: * Scaling to "In-the-Wild" Video: Extending the model to learn from arbitrary YouTube or internet videos of humans. * Online World Models: Currently, the world model is frozen during policy improvement. Updating the world model with real-world rollouts from the fine-tuned policy could further improve performance. * Reducing Robot Data Further: Investigating the absolute minimum amount of robot play needed to ground the latent actions effectively.
Summary updated