DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play

Akshay L Chandra, Iman Nematollahi, Lingyue Zhao, Daria Kulikova, Angela P. Schoellig, Fabien Despinoy, Abhinav ValadaarXiv 2026 · arXiv

TL;DR

DiWALI is a novel framework that enables robots to learn complex manipulation skills from action-free human videos by inferring "latent actions" between frames. By using these latent actions to build a world model, the system can train and refine diffusion policies entirely in "imagination," significantly outperforming traditional methods that rely on scarce, action-labeled robot data.

What Problem Does This Paper Address?

The "robot-time bottleneck" is a major hurdle in scaling robot learning. Collecting high-quality robot data usually requires teleoperation, where one hour of data takes one hour of human and robot time. In contrast, humans can demonstrate tasks in seconds, and the internet is full of human videos.

However, these human videos lack action labels—the specific motor commands (joint torques or end-effector velocities) a robot needs. Furthermore, there is an embodiment gap: a human hand looks and moves differently than a robotic gripper. Most current world models (which act as internal simulators for robots) require action labels to learn the underlying physics. DiWALI aims to bridge this gap, allowing robots to learn from "play" data and demonstrations of humans without needing any robot-specific action signals.

Methods

The DiWALI (Diffusion Policy Learning in World Models via Latent Action Inference) framework operates in four distinct stages:

  1. Latent-Action World Model: Instead of conditioning a world model on robot commands, the authors use an inverse dynamics model to infer a "latent action" ($u_t$) from two consecutive video frames. This latent action captures the effect of a movement regardless of whether a human or a robot performed it. This allows a Recurrent State-Space Model (RSSM) to learn scene dynamics from both human and robot "play" data.
  2. Action Decoder: To make these latent actions executable, a small decoder is trained separately on a small amount of robot play data. It learns to map a latent action and the current state to a real robot command.
  3. Action-Free Policy Learning: A diffusion policy is trained to imitate human demonstrations. Because the policy operates in the latent action space, it can learn directly from human videos without needing to know the underlying motor commands.
  4. Policy Improvement in Imagination: The learned world model serves as a simulator. The policy is fine-tuned using Reinforcement Learning (PPO) within this "imagination" space, allowing the robot to improve its skills without further real-world interaction.
Fig. 2: DiWALI framework. (a) A latent-action world model is trained on robot and human play, where an inverse dynamics model infers a latent action ut−1 from the embeddings of two consecutive observations, and the RSSM uses it in place of the robot command to advance its recurrent state, predict the prior ˆzt, and reconstruct the observation. No action labels are used. (b) The frozen world model encodes robot play into latent states zt and latent actions ut, from which an action decoder learns to predict the executed robot commands at. (c) Human demonstrations are encoded the same way, and a diffusion policy learns to denoise Gaussian noise into their latent actions conditioned on zt. (d) The policy is fine-tuned with PPO on imagined rollouts, in which the frozen world model advances the latent state under the policy’s latent actions, and a frozen reward classifier scores each state. Snowflakes mark frozen components, flames trainable ones.

Datasets

  • CALVIN: A standard benchmark for long-horizon robot manipulation.
  • CALVIN-H: A new dataset introduced by the authors where the robot in CALVIN is replaced by a rendered human arm, and all action labels are removed to simulate human video data.
  • Real-World Data: The authors collected 2 hours of robot play and 2 hours of human play (using a wrist-mounted camera to match the robot's perspective) in a tabletop environment.

Key Insights and Ideas

  • Latent Actions as a Universal Language: By inferring actions from visual changes rather than motor commands, the model creates a shared representation for different embodiments. This effectively bypasses the need for manual retargeting of human motions to robot joints.
  • The Importance of State-Conditioned Decoding: A key "aha moment" in the paper is that a latent action (e.g., "move right") requires different motor commands depending on the robot's current configuration. Conditioning the action decoder on the latent state was critical, nearly tripling success rates in ablations.
  • Separate vs. Joint Training: The authors found that training the action decoder after the world model is frozen is superior to joint training. Joint training tends to "collapse" the latent space toward only what the robot can do, losing the rich diversity of movement found in human data.
  • Human Play as a Simulator Booster: Including human play data doesn't just help the policy; it makes the world model a better simulator. This leads to more robust fine-tuning in imagination compared to using robot data alone.

Key Results

DiWALI demonstrates a significant leap in data efficiency:

  • Outperforming Baselines: On the CALVIN benchmark, a DiWALI policy trained with zero robot demonstrations achieved a 59.6% success rate, outperforming a standard Diffusion Policy trained with five robot demonstrations (35.2%).
  • Imagination Works: In real-world experiments, skills learned primarily from human videos improved from 53% to 70% success after fine-tuning in the world model's imagination, with no additional physical robot time required.
  • Ablation Success: The choice to condition the decoder on the latent state and train it separately was the most impactful design decision, raising success from 18.7% to 61.6% in controlled tests.
Visualizing imagined rollouts: The world model successfully predicts futures for both robot and human embodiments based on inferred latent actions.

Limitations and Future Work

While DiWALI reduces the need for action labels, it still requires some robot play data to ground the action decoder. The authors also note that the current model is trained on human data collected in the same workspace as the robot.

Future directions include: * Scaling to "In-the-Wild" Video: Extending the model to learn from arbitrary YouTube or internet videos of humans. * Online World Models: Currently, the world model is frozen during policy improvement. Updating the world model with real-world rollouts from the fine-tuned policy could further improve performance. * Reducing Robot Data Further: Investigating the absolute minimum amount of robot play needed to ground the latent actions effectively.

Summary updated

Abstract

Learning robot manipulation is constrained by the availability of action-labeled robot data, for both imitation and world models. Humans can demonstrate skills in seconds, but their demonstrations lack the required robot actions. We introduce DiWALI, a framework that learns and improves diffusion policies within a world model trained on human and robot play. DiWALI infers latent actions from consecutive frames using an inverse dynamics model, enabling world model training without action labels, and grounds them in robot commands through a decoder trained separately on robot play. Skill policies are trained over latent actions from action-free human demonstrations and improved with reinforcement learning in imagination. On the CALVIN benchmark, DiWALI policies trained solely from human demonstrations substantially outperform baselines trained with robot demonstrations. Incorporating human play into the world model training also yields stronger policies after fine-tuning than using the same amount of robot play alone. Experiments on a physical robot show that skills learned primarily from human demonstrations are further improved in imagination with a world model trained on human and robot play. Together, these results demonstrate that human experience can support both manipulation skill acquisition and subsequent policy improvement, reducing reliance on action-labeled robot data. Videos, dataset and code are available at https://diwali.cs.uni-freiburg.de.
BibTeX
@misc{chandra2026diwali,
  author = {Akshay L Chandra and Iman Nematollahi and Lingyue Zhao and Daria Kulikova and Angela P. Schoellig and Fabien Despinoy and Abhinav Valada},
  title = {{DiWALI: Diffusion Policy Learning in World Models via Latent Action Inference from Human and Robot Play}},
  year = {2026},
  eprint = {2611.24836},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24836},
}
Figure