TL;DR
pico-WAM is a compact diagnostic architecture designed to pinpoint why world models fail at robot control: is it because they can't propose good actions, or because they can't accurately score the outcomes? The study reveals that trajectory scoring is the primary bottleneck; while training on counterfactual "off-policy" data significantly improves ranking, replacing learned scorers with ground-truth simulator costs provides a massive performance boost that improved proposal generation cannot match.
What Problem Does This Paper Address?
World Action Models (WAMs) promise to let robots "think before they act" by imagining future states. However, when these models fail in closed-loop control, researchers often struggle to identify the culprit. Does the model fail to suggest a winning action sequence (proposal generation), or does it incorrectly predict that a bad action will lead to a good result (trajectory scoring)?
Most modern WAMs are complex, end-to-end systems where these components are tightly coupled. This paper introduces pico-WAM, a "transparent" and compact setup designed to isolate these interactions. It specifically tackles the "off-manifold" problem: models trained only on expert demonstrations often have no idea what happens when they deviate from the perfect path, leading to catastrophic errors during planning.
Methods
The authors use a shared representation architecture where a single latent state $z_t$ feeds three distinct heads:
- Policy ($\pi$): Proposes "what action should I imitate?"
- Dynamics ($f_\theta$): Predicts "what happens if I take this action?"
- State Readout ($R$): Maps latents back to physical coordinates for transparency.
Key methodological innovations include:
- Branch Training: Instead of training only on expert demonstrations, the authors use a simulator to create "counterfactual branches"—perturbing expert actions to see what happens when the robot makes a mistake.
- Crossed Factorial Evaluation: To isolate errors, they perform "surgery" on the planner, swapping learned components with "Exact" (simulator-based) components. They test four combinations: Learned Proposals/Learned Scoring (LL), Learned/Exact (LE), Exact/Learned (EL), and Exact/Exact (EE).
- Search Strategies: They compare Warm-starting (refining the previous plan) against Restarting (starting fresh from the policy's suggestion at every step).
Datasets
- Push-T: The primary benchmark, involving pushing a T-shaped block onto a target. The authors used 206 demonstrations from the LeRobot dataset.
- Meta-World (door-open-v3): A supplementary study used to verify if the benefits of broader transition data (branch training) generalize to different architectures and tasks.
Key Insights and Ideas
- Scoring is the Bottleneck: The most striking finding is that improving the scoring of plans is far more effective than improving the proposals. If you give the model perfect scoring (via the simulator) but keep its learned proposals, performance sky-rockets. If you do the opposite, there is no detectable gain.
- The "Optimism" Trap: Models are systematically optimistic. As shown in Figure 4, models often predict significant progress (e.g., moving a block 10px) but realize almost none in reality. Branch training helps mitigate this by teaching the model about failure.
- Restarting Beats Refining: In compact models, "Warm-starting" a search can actually be a trap. If the previous plan was based on a hallucination, refining it just polishes a mistake. Restarting from the policy at each step proved more robust.
- Ranking $\neq$ Success: A model might get very good at ranking a fixed set of 256 plans (improving rank correlation from 0.49 to 0.80), but this doesn't guarantee it will actually complete the task. Closed-loop control introduces compounding errors that simple ranking tests don't capture.
Key Results
- Branch Training Impact: Adding counterfactual branches raised task coverage from 43.8% to 78.4% when using policy-restarted search.
- The Scoring Boost: Replacing learned trajectory scoring with ground-truth simulator costs boosted coverage by 12.2 percentage points while retaining learned proposals.
- Proposal Substitution: Substituting simulator states during proposal generation alone showed no detected gain, reinforcing that the model's "imagination" of actions was already sufficient; its "judgment" of outcomes was the weak link.
Limitations and Future Work
- Compactness vs. Scale: The study uses a compact MLP-based architecture. While this allows for clear diagnostics, it remains to be seen if these specific bottlenecks (scoring vs. proposal) shift as models scale to massive Transformers.
- State Dependency: The current model relies on supplied physical state at inference time. Future work could explore how these dynamics change when the model must rely entirely on raw visual latent representations.
- The Gap to Imitation: Despite the planning improvements, the best planners often only matched or slightly exceeded well-tuned imitation learning (BC) baselines. The authors suggest that identifying which specific contact or state errors drive remaining failures is the next frontier for world model research.
Summary updated