TL;DR
Doppel is a video world model that predicts future scenes by conditioning on image-space renderings of a robot's intended motion. By offloading the "solved" problem of kinematics to a renderer, the model can focus its capacity on complex environment dynamics, allowing a single shared model to generalize across diverse robots and even human embodiments with state-of-the-art accuracy.
What Problem Does This Paper Address?
Predicting the future is hard, especially for a robot. Traditional video world models take an initial image and a sequence of numerical action vectors (like joint velocities) to "imagine" what happens next. This approach forces the model to learn two distinct, difficult mappings simultaneously:
- Kinematics: How do these numbers translate into the visual movement of my own body?
- Dynamics: How does my body's movement affect the objects around me?
Furthermore, because every robot has a different "language" of action vectors (different joint counts, ranges, and geometries), it is notoriously difficult to train a single world model that works across multiple different robot types.
Methods
The authors introduce Doppel, which simplifies the learning process by providing the model with a "cheat sheet" of its own intended motion.
The Core Idea: Visual Conditioning
Instead of just giving the model numbers, Doppel uses the robot's known geometry (URDF) and forward kinematics to render a "ghost" of the robot's future poses directly into the image space. This provides a common visual interface: whether the agent is a Franka arm, a humanoid, or a human hand, the model sees a visual representation of where that body will be.
Architecture
Doppel builds on a pre-trained video generation backbone (Wan 2.1). It uses a novel Rendering-Enriched Action Conditioning Adapter to fuse three inputs:
- Initial Image: The starting state of the world.
- Future Renderings: A sequence of images showing only the robot moving as commanded.
- Numerical Actions: Retained to capture subtle details that might be ambiguous in a 2D render.
Camera Calibration at Scale
A major hurdle for this approach is that many existing robot datasets lack precise camera "extrinsics" (the exact position of the camera relative to the robot). The authors developed an automated pipeline that uses a differentiable renderer to "solve" for the camera pose by matching rendered robot motion to the actual video, enabling them to use massive amounts of existing data.
Datasets
The model was trained on a massive corpus of ~600 hours of video across eight distinct embodiments:
- Robot Arms: DROID (Franka), Bridge (WidowX), and RoboMIND (UR5e).
- Bimanual/Humanoid: ABC (YAM bimanual) and T-Rex (humanoid with dexterous hands).
- Human Motion: Ego-Exo4D (full body), GigaHands (hands), and OakInk2 (upper body).
Key Insights and Ideas
- Factorization is Power: By explicitly providing the "intended" motion, the model doesn't waste parameters learning basic kinematics. It can focus entirely on the "unsolved" part: the physics of interaction.
- Pixels as a Universal Language: Renderings provide a "shared embodiment" space. A model doesn't need to know it's looking at a "WidowX" robot; it just sees pixels moving and learns how the environment reacts to those specific pixel-movements.
- Cross-Embodiment Synergy: Surprisingly, training on human data and diverse robots (Doppel-8) actually improved the model's performance on specific robot tasks compared to training on robots alone (Doppel-3). The model learns a more general "common sense" about physics.
Key Results
Doppel significantly outperforms standard action-conditioned baselines:
- Higher Fidelity: On three major robot datasets, Doppel improved PSNR (a measure of image reconstruction quality) by 1.00 dB on unseen trajectories and 1.20 dB on out-of-distribution (OOD) scenes.
- Efficiency: Doppel achieved better results than a shared action-only baseline while using 63.8% fewer training draws.
- Better Physics: Qualitatively, Doppel is much better at predicting object interactions. While baselines often "ghost" the robot through objects, Doppel correctly predicts objects sliding, rotating, or being lifted.
Limitations and Future Work
- Calibration Dependency: The method relies on having an accurate URDF and knowing the camera pose. It struggles if the camera moves unexpectedly or if the robot's geometry is unknown.
- Proprioception Only: The model assumes the robot follows its commanded trajectory perfectly. It doesn't yet account for "compliance" (the robot being pushed back by a heavy object).
- Zero-Shot Transfer: While the model generalizes well, the authors leave the challenge of transferring to a completely unseen robot geometry (without any fine-tuning) for future research.
Summary updated