Search or Scoring? Diagnosing Why Visual World-Model Planners Fail

Haoxuan Jia, Yang Liu, Shang Luo, Chenhao Wu, Zixiang Ni, Yingguang Yang, Feng Zhang, Chongyang Zhang, Hao Zheng, Kefu Xu, Congjing Ran, Jifeng Zhu, Bin ChongarXiv 2026 · arXiv

TL;DR

Visual world-model planners fail for two distinct reasons: either the optimizer misses a good action (search failure) or the model incorrectly overrates a bad one (scoring failure). This paper introduces a simple diagnostic probe to distinguish these failure modes and proposes DemoProp, a method that uses retrieved demonstrations to either "warm-start" the search or bypass the model entirely by averaging demonstration windows, significantly improving success rates across diverse robotic tasks.

What Problem Does This Paper Address?

Visual world models (like DINO-WM and JEPA-WM) allow robots to plan complex tasks without explicit rewards by imagining future outcomes and selecting actions that lead to a goal image. However, these planners often fail. Traditionally, researchers respond by simply "searching more" (increasing compute) or "improving the model."

The authors argue that these blanket responses are inefficient because they ignore the root cause of failure. A planner can fail in two fundamentally different ways:

  1. Search Failure: The model actually knows what a good action looks like, but the optimizer (usually the Cross-Entropy Method, or CEM) simply never sampled it.
  2. Scoring Failure: The model is overconfident in its own errors. It assigns a lower cost to a failing action than to a successful one, essentially "tricking" the optimizer.

Without knowing which failure occurred, you don't know whether to add more compute or fix the model's overestimation.

Methods

The authors propose a known-solution probe to diagnose failures. By taking a known successful action sequence (from a demonstration) and scoring it with the planner's own cost function, they can compare it to the optimizer's best candidate.

  • If the model prefers the known solution but the optimizer didn't find it, it's a search failure.
  • If the model prefers its own failing candidate over the known solution, it's a scoring failure.

To fix these, they introduce DemoProp (Demonstration Propagation), which retrieves relevant "windows" of actions from a demonstration library based on the robot's current state and goal.

  • In the Search Regime: They use a "warm start," seeding the optimizer with these retrieved demonstrations.
  • In the Scoring Regime: Since the model's judgment is untrustworthy, they use "Average"—simply executing the mean of the retrieved demonstration windows, bypassing the model's faulty scoring.
Figure 1. Two ways a world-model planner fails. Left, a search failure on PushT (development task): the model’s cost for the known solution (0.091) is far below that of the best of the 900 candidates recorded from the cross-entropy method (CEM; 0.33), and adding retrieved demonstrations to the candidates (Expand) lets the cost pick one that works. Right, a scoring failure on Metaworld door-open: the model assigns CEM’s action a lower cost than the known solution and predicts it to open the door (decoded prediction), yet the door stays half open; Average, the mean of the 30 nearest demonstration windows, opens it. Dashed outline: the block’s goal pose. ✓/×: full success.

Datasets

The study evaluates three world models across four simulated environments:

  • PushT: A 2D planar task involving pushing a T-shaped block to a goal pose.
  • Wall: A 2D navigation task requiring an agent to pass through a specific door.
  • Metaworld: A 3D manipulation benchmark (e.g., opening a faucet, sliding a plate).
  • RoboCasa: A photorealistic kitchen simulator using models trained on real-robot data (DROID dataset).

Key Insights and Ideas

  • The Optimizer's Curse: In the scoring regime, searching more can actually be harmful. A larger search is more likely to find "adversarial" actions that exploit the model's prediction errors, leading to even lower predicted costs for actions that fail in reality.
  • Environment-Specific Regimes: PushT and Wall are primarily limited by search. Metaworld and RoboCasa are dominated by scoring failures. Interestingly, the cost function is often "sound" on real outcomes (high AUROC) but "at chance" on the model's own noisy predictions.
  • The Power of Averaging: When a model is in the scoring regime, the best move is often to ignore its selection entirely. Averaging nearby demonstrations cancels out individual "scatter" in the data without falling into the trap of the model's overestimation.
Figure 2. The known-solution probe results. Dots below 1 indicate search failures (PushT, Wall), while dots above 1 indicate scoring failures (Metaworld, RoboCasa).

Key Results

  • Search Regime (PushT): Using DemoProp to expand the candidate pool raised the success rate of DINO-WM from 62 to 78 out of 96 episodes.
  • Scoring Regime (Metaworld): On 168 new tasks, the "Average" method solved 127 tasks, whereas letting the model choose among the same windows solved at most 87.
  • Diagnostic Robustness: The authors developed a heuristic rule to choose between "warm start" and "average" based on the failure regime. This rule correctly adapted even when the model was artificially degraded (quantized), solving 678 of 960 new PushT episodes compared to 530 for the best fixed strategy.
Figure 3. DemoProp. (1) Query: the pusher now and at the goal. (2) Four of the 100 retrieved windows (PushT task 51). (3) Search regime (official PushT episode 0, DINO-WM): CEM started from the windows finds a plan with a lower predicted cost than the known solution, and the block ends 3.6 units from its goal on the 512-unit canvas, against 75 from random samples. Scoring regime (Metaworld pick-out-of-hole, new task): the mean of the 30 nearest windows leaves the object 3.1 cm from its goal, the window ranked first 14 cm. ✓/×: full success.

Limitations and Future Work

  • Requirement for Demonstrations: The diagnostic and the remedy both rely on having access to a library of successful demonstrations.
  • Short Horizons: The study focuses on short-horizon "window tasks." Extending this diagnosis to long-horizon planning where errors compound is a major next step.
  • Simulation vs. Reality: While RoboCasa uses real-robot data, all evaluations were performed in simulation. Testing the "Average" remedy on physical hardware where safety and precision are paramount remains future work.

Summary updated

Abstract

Visual world models such as DINO-WM and JEPA-WM plan with an optimizer that searches for actions whose predicted outcome has the lowest cost, its distance to a goal image. A plan can thus fail in two ways: the optimizer never tries a good action that the cost would prefer (a search failure), or the cost, computed on predictions, ranks a bad action first (a scoring failure). The two need different remedies, more search or a better model, but a planner does not record which one occurred. We tell them apart by executing the optimizer's candidates, scoring a known solution with the planner's own cost, and comparing costs on predicted and real outcomes. An error model explains when each arises and so how to use retrieved demonstration windows (DemoProp): start the search from them (a warm start) when prediction errors are small, and average them without the model when large errors make the optimizer pick overrated actions. Across four simulated environments and three world models, on short tasks whose goals the planner can represent, PushT and Wall fail mostly by search, while Metaworld and RoboCasa fail by scoring because the cost, sound on real outcomes (AUROC 0.94 on Metaworld), is at chance on predictions (0.54). The diagnosis tells which remedy pays off: a four times larger search lowers the real-outcome cost only where failures come from search (clearly for DINO-WM), and where they come from scoring, averaging the windows without the model solves 127 of 168 new Metaworld tasks, against at most 87 when the model chooses among them. Within PushT, degrading only the model flips which use is better, and our rule, frozen in advance, follows the flip and solves 678 of 960 new episodes, against 530 for the best fixed use.
BibTeX
@misc{jia2026search,
  author = {Haoxuan Jia and Yang Liu and Shang Luo and Chenhao Wu and Zixiang Ni and Yingguang Yang and Feng Zhang and Chongyang Zhang and Hao Zheng and Kefu Xu and Congjing Ran and Jifeng Zhu and Bin Chong},
  title = {{Search or Scoring? Diagnosing Why Visual World-Model Planners Fail}},
  year = {2026},
  eprint = {2611.25347},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25347},
}
Figure