DRILL: Training World Models to Improve Policies, Not Predict Pixels

Zhaolu Kang, Tailong Luo, Lei Wei, Guangyuan Dong, Sihan Liu, Yingjie He, Siheng Wang, Shuaibo Li, Rongchao Zhang, Zhen Tian, Richeng Xuan, Zhichao HuarXiv 2026 · arXiv

TL;DR

Standard world models for robotics are trained to be great video predictors, but being a good "artist" doesn't make you a good "teacher." DRILL (Downstream-Return Imagination Learning Loop) shifts the training objective from minimizing pixel error to maximizing the success of the policy trained within the model, leading to significant gains in robot manipulation tasks even when visual fidelity decreases.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) policies are the "brains" of modern robots, but training them directly in the real world is slow and expensive. Researchers often use world models—generative models that act as virtual simulators—to let policies practice in "imagination."

The catch? These world models are almost always trained to minimize pixel-level Mean Squared Error (MSE). This creates an objective mismatch. A model might spend all its capacity perfectly rendering a static kitchen background (low MSE) while failing to accurately simulate the physics of a small, task-critical handle. In other words, a model can be an excellent next-frame predictor while producing weak or misleading learning signals for the robot.

Methods

The authors propose DRILL, a bilevel optimization framework. Instead of just predicting the next frame, the world model is optimized based on how much the policy improves after "dreaming" inside it.

Figure 1: Overview of DRILL. The policy first learns inside the current world model through imagined GRPO. The resulting policy-improvement signal is then fed back to the world model: DRILL-VWI uses the same imagined rollouts to form a signed, visitation-weighted local surrogate, whereas DRILL-IMG differentiates through the inner update and grounds the outer gradient with a small simulator batch.

DRILL comes in two flavors:

  1. DRILL-VWI (Visitation-Weighted Imagination): A "budget-conscious" version. It doesn't require extra simulator runs. Instead, it reweights standard pixel errors using three factors already available during training: where the policy goes (visitation), which actions were successful (advantage), and how sure the model is (confidence).
  2. DRILL-IMG (Implicit Meta-Gradient): The "high-fidelity" version. It uses a small batch of real simulator rollouts to calculate a "meta-gradient." It essentially asks: "How should I change the world model's parameters so that the policy trained on its output performs better in the real world?" This involves differentiating through the entire policy update using truncated Hessian-vector products.

The authors also introduce a new diagnostic metric, the Policy-Utility Residual (PUR), which measures the alignment between the pixel-prediction gradient and the actual policy-improvement direction.

Datasets

The researchers evaluated DRILL across five diverse robot manipulation simulators and two major VLA backbones ($\pi_0$-3B and OpenVLA-7B):

  • LIBERO-Long: A suite of long-horizon manipulation tasks.
  • CALVIN: Language-conditioned manipulation focusing on cross-environment generalization.
  • RoboCasa: Large-scale kitchen simulation.
  • Meta-World MT50: A standard multi-task manipulation benchmark.
  • SimplerEnv: A simulated evaluation suite for VLA policies.

Key Insights and Ideas

  • The MSE-Success Decoupling: One of the most striking findings is that visual realism and task utility are not the same thing. Models trained with DRILL often have higher pixel MSE (they look "worse" to the human eye) but produce vastly superior robot policies.
  • Focusing on the "Interesting" Pixels: By weighting the loss by policy advantage, DRILL forces the world model to prioritize accuracy on transitions that actually matter for succeeding at the task, rather than wasting capacity on task-irrelevant background details.
  • Horizon Sensitivity: The benefit of the full meta-gradient (DRILL-IMG) grows with the task's complexity. For short tasks, the simpler VWI surrogate is often enough. For long-horizon tasks where errors compound, the grounded meta-gradient becomes essential.
Figure 2: MSE–success decoupling on LIBERO-Long (π0-3B). Faint markers are seeds; bold markers are mean±std.

Key Results

  • Performance Boost: On the $\pi_0$-3B backbone, DRILL-IMG raised the mean success rate by 13.1 points over standard WMPO and 7.9 points over RLVR-World.
  • Efficiency: This improvement was achieved even though the world model had 17% higher pixel MSE on the same training data.
  • Diagnostic Validation: The PUR metric proved to be a reliable "smoke test"—lower PUR values (better alignment) consistently correlated with higher downstream success rates.
  • Human Perception vs. Utility: In a blinded study, humans couldn't statistically distinguish the visual quality of DRILL-generated videos from baselines. However, when shown the final task success, they preferred DRILL-trained models by a wide margin (62% vs 15%).
Figure 3: Three compact diagnostics on LIBERO-Long. (a) DRILL variants pass baselines early and continue improving. (b) Larger rollout horizon widens the DRILL-IMG–DRILL-VWI gap until imagined trajectories become noisy. (c) Lower PUR tracks higher policy utility during training.

Limitations and Future Work

  • Simulation Only: The current study is entirely simulation-based. While the simulators are high-fidelity, the "sim-to-real" gap remains a hurdle for future research.
  • Computational Cost: DRILL-IMG is more computationally expensive than standard training, requiring more memory and time due to the meta-gradient calculations.
  • Safety and Constraints: The paper does not explicitly address real-world safety constraints or actuation delays that might appear during physical robot deployment.

The authors suggest that the next logical steps are extending DRILL to real-robot post-training and finding ways to further reduce the computational overhead of the meta-gradient estimator.

Summary updated

Abstract

World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically trained to predict pixels one step ahead, even though their role here is not merely to render visually faithful videos but to produce learning signals from imagined rollouts that improve the policy. We show that this mismatch is substantial: lower pixel MSE can produce worse policy gradients, and a world model trained for policy utility can yield better downstream policies despite 17% higher pixel MSE on the next frame. We propose the Downstream-Return Imagination Learning Loop (DRILL), a bilevel framework that updates the world model to maximize the policy's return after an inner GRPO update on imagined rollouts. DRILL has two instantiations. DRILL-VWI is a closed-form surrogate, requiring no additional simulator rollouts, that reweights prediction errors by policy visitation, the group-relative GRPO advantage, and model confidence. DRILL-IMG is a full meta-gradient method that differentiates through the inner update using truncated Hessian-vector products. We show that DRILL-VWI recovers the local one-step component of the DRILL-IMG meta-gradient under score-compatibility and small-residual assumptions. Across five manipulation simulators and two VLA backbones, DRILL-IMG raises mean success rate by 13.1 points over WMPO and 7.9 points over RLVR-World, and complements RLVR pretraining rather than competing with it. More broadly, we believe video world models for VLA reinforcement learning can be productively trained by the policy gradients they produce, not only by pixel-level prediction.
BibTeX
@misc{kang2026drill,
  author = {Zhaolu Kang and Tailong Luo and Lei Wei and Guangyuan Dong and Sihan Liu and Yingjie He and Siheng Wang and Shuaibo Li and Rongchao Zhang and Zhen Tian and Richeng Xuan and Zhichao Hu},
  title = {{DRILL: Training World Models to Improve Policies, Not Predict Pixels}},
  year = {2026},
  eprint = {2610.30814},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.30814},
}
Figure