Render the Robot, Generate the World: One Video World Model Across Embodiments

Yash Jangir, Karan Mirakhor, Renos Zabounidis, Muhammad Zubair Irshad, Sergey Zakharov, Yonatan Bisk, Homanga BharadhwajarXiv 2026 · arXiv

TL;DR

Doppel is a video world model that predicts future scenes by conditioning on image-space renderings of a robot's intended motion. By offloading the "solved" problem of kinematics to a renderer, the model can focus its capacity on complex environment dynamics, allowing a single shared model to generalize across diverse robots and even human embodiments with state-of-the-art accuracy.

What Problem Does This Paper Address?

Predicting the future is hard, especially for a robot. Traditional video world models take an initial image and a sequence of numerical action vectors (like joint velocities) to "imagine" what happens next. This approach forces the model to learn two distinct, difficult mappings simultaneously:

  1. Kinematics: How do these numbers translate into the visual movement of my own body?
  2. Dynamics: How does my body's movement affect the objects around me?

Furthermore, because every robot has a different "language" of action vectors (different joint counts, ranges, and geometries), it is notoriously difficult to train a single world model that works across multiple different robot types.

Methods

The authors introduce Doppel, which simplifies the learning process by providing the model with a "cheat sheet" of its own intended motion.

Figure 1: Doppel is a surprisingly simple approach for incorporating future embodiment renderings and improving robot-object interactions in action-conditioned video world models.

The Core Idea: Visual Conditioning

Instead of just giving the model numbers, Doppel uses the robot's known geometry (URDF) and forward kinematics to render a "ghost" of the robot's future poses directly into the image space. This provides a common visual interface: whether the agent is a Franka arm, a humanoid, or a human hand, the model sees a visual representation of where that body will be.

Architecture

Doppel builds on a pre-trained video generation backbone (Wan 2.1). It uses a novel Rendering-Enriched Action Conditioning Adapter to fuse three inputs:

  • Initial Image: The starting state of the world.
  • Future Renderings: A sequence of images showing only the robot moving as commanded.
  • Numerical Actions: Retained to capture subtle details that might be ambiguous in a 2D render.
The Doppel architecture fuses VAE-encoded renderings with numerical actions to modulate a pre-trained video backbone.

Camera Calibration at Scale

A major hurdle for this approach is that many existing robot datasets lack precise camera "extrinsics" (the exact position of the camera relative to the robot). The authors developed an automated pipeline that uses a differentiable renderer to "solve" for the camera pose by matching rendered robot motion to the actual video, enabling them to use massive amounts of existing data.

Datasets

The model was trained on a massive corpus of ~600 hours of video across eight distinct embodiments:

  • Robot Arms: DROID (Franka), Bridge (WidowX), and RoboMIND (UR5e).
  • Bimanual/Humanoid: ABC (YAM bimanual) and T-Rex (humanoid with dexterous hands).
  • Human Motion: Ego-Exo4D (full body), GigaHands (hands), and OakInk2 (upper body).

Key Insights and Ideas

  • Factorization is Power: By explicitly providing the "intended" motion, the model doesn't waste parameters learning basic kinematics. It can focus entirely on the "unsolved" part: the physics of interaction.
  • Pixels as a Universal Language: Renderings provide a "shared embodiment" space. A model doesn't need to know it's looking at a "WidowX" robot; it just sees pixels moving and learns how the environment reacts to those specific pixel-movements.
  • Cross-Embodiment Synergy: Surprisingly, training on human data and diverse robots (Doppel-8) actually improved the model's performance on specific robot tasks compared to training on robots alone (Doppel-3). The model learns a more general "common sense" about physics.

Key Results

Doppel significantly outperforms standard action-conditioned baselines:

  • Higher Fidelity: On three major robot datasets, Doppel improved PSNR (a measure of image reconstruction quality) by 1.00 dB on unseen trajectories and 1.20 dB on out-of-distribution (OOD) scenes.
  • Efficiency: Doppel achieved better results than a shared action-only baseline while using 63.8% fewer training draws.
  • Better Physics: Qualitatively, Doppel is much better at predicting object interactions. While baselines often "ghost" the robot through objects, Doppel correctly predicts objects sliding, rotating, or being lifted.
Doppel preserves consistent motion and predicts plausible environment responses even in out-of-domain scenes.

Limitations and Future Work

  • Calibration Dependency: The method relies on having an accurate URDF and knowing the camera pose. It struggles if the camera moves unexpectedly or if the robot's geometry is unknown.
  • Proprioception Only: The model assumes the robot follows its commanded trajectory perfectly. It doesn't yet account for "compliance" (the robot being pushed back by a heavy object).
  • Zero-Shot Transfer: While the model generalizes well, the authors leave the challenge of transferring to a completely unseen robot geometry (without any fine-tuning) for future research.

Summary updated

Abstract

Predicting the consequences of an agent's actions requires modeling both the body's motion and the environment's response. Most video world models attempt to learn both from numerical actions, even though geometry and kinematics already specify the intended motion. We introduce Doppel, an embodiment-agnostic video world model conditioned on camera-aligned renderings of future body configurations and numerical action trajectories. By explicitly providing the intended motion, Doppel can focus on predicting object motion and interaction-driven changes in the environment. Rendered motion also provides a common visual interface across different embodiments and action spaces. To train on robot datasets without camera extrinsics, we recover camera poses and align rendered trajectories with recorded observations. We jointly train a single shared video backbone and conditioning adapter on eight datasets spanning eight robot and human embodiments, from single- and bimanual robot arms to a dexterous humanoid and human hands and bodies, using about 600 hours of video without embodiment labels. Across three robot-arm datasets, Doppel improves PSNR over action-only baselines by 1.00 dB on held-out trajectories and 1.20 dB under out-of-distribution evaluation, while predicting more consistent object motion. Training on all eight embodiments (Doppel-8) improves robot predictions over the three-robot model (Doppel-3) on four of six evaluation sets. Project page: https://doppelwm.github.io/.
BibTeX
@misc{jangir2026render,
  author = {Yash Jangir and Karan Mirakhor and Renos Zabounidis and Muhammad Zubair Irshad and Sergey Zakharov and Yonatan Bisk and Homanga Bharadhwaj},
  title = {{Render the Robot, Generate the World: One Video World Model Across Embodiments}},
  year = {2026},
  eprint = {2611.24925},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24925},
}
Figure