TL;DR
This paper reveals that standard methods for evaluating "Best-of-N" selection in World Action Models (WAMs) are heavily biased by future randomness, a phenomenon known as the "optimizer's curse." By introducing a denoised counterfactual protocol, the authors show that real "selectable headroom" comes from structured action perturbations rather than native policy samples, and they leverage this to build a fail-gated selector that significantly improves closed-loop robot task success.
What Problem Does This Paper Address?
World Action Models (WAMs) are exciting because they don't just predict actions; they predict the visual consequences of those actions. This enables test-time selection: a robot can "imagine" several possible futures for different candidate actions and execute the one that looks most successful.
However, how much can this actually help? Previous studies measured "oracle gain" by executing multiple candidates once and picking the best outcome. The authors argue this is a flawed metric. Because the future is stochastic, the candidate that succeeds in a single trial might just be the "luckiest" one, not the best one. This conflation of candidate quality with continuation randomness leads to an overestimation of the selectable headroom—the actual performance gain a real-world selector could achieve.
Methods
The authors propose a denoised counterfactual protocol to isolate true candidate quality from noise. Key components include:
- Bitwise-Reproducible Branching: Using simulator snapshots (MuJoCo) to ensure that every candidate action starts from the exact same state, down to the last bit of the random number generator.
- Repeated Continuations: Instead of running a candidate once, they run it multiple times ($J$ continuations) with different future seeds.
- Held-out Oracles: They use a "double estimator" approach—selecting the best candidate based on one set of continuations and scoring it on a completely fresh, held-out continuation. This eliminates the "optimizer's curse."
- Fail-Gated Selector: They design a practical selector that uses a failure gate to decide when to deviate from the base policy (only when failure is predicted) and a ranker to decide where to go among structured perturbations (e.g., shifting the action along coordinate axes).
Datasets
The study evaluates four different WAMs (Cosmos Policy, Flex-π, Fast-WAM, and Streaming-WAM) across two major benchmarks:
- RoboCasa: A large-scale simulation for everyday kitchen tasks.
- LIBERO-Plus: A benchmark focused on lifelong learning and robustness to object layouts and robot initial states. The authors also release HeadroomBench, a package containing their protocol and over 100,000 branched rollouts for future research.
Key Insights and Ideas
- The "Optimizer's Curse" is Real: In native policy samples, up to 98% of the variation between candidates is just "continuation luck." Standard estimates overstate the selectable headroom by 1.5x to 4x.
- Native Samples vs. Structured Perturbations: Sampling more "native" actions from the policy provides almost no real headroom because the samples are too similar. However, structured perturbations (e.g., shifting the gripper position by 2-10cm) create significant, realizable headroom, especially in states where the base policy is likely to fail.
- Value Heads are Blind to Sibling Actions: Standard value heads are great at judging if a state is good, but they are surprisingly bad at ranking "sibling" actions from the same state. They miss the fine-grained signal present in the model's own imagined futures.
- Imagination is High-Fidelity: Probes show that the information needed to rank actions is present in the WAM's imagined futures with no detectable loss compared to "real" futures. The bottleneck isn't the world model's prediction; it's the readout (the value head).
Key Results
- Closed-Loop Success: The fail-gated selector improved task success from 76.9% to 81.6% on training tasks and from 62.6% to 69.8% on entirely unseen tasks.
- Targeted Gains: The largest improvements occurred exactly where the policy was weakest. In "robot initial state" perturbations, success jumped by 10.7 points.
- Efficiency: By using a failure gate, the selector only triggers expensive "imagination" passes at about 35-45% of decision points, making it much more computationally efficient than a naive Best-of-N approach.
Limitations and Future Work
- Simulation vs. Reality: The study relies on bitwise-reproducible branching in simulation. While the selector doesn't need this at deployment, the measurement of headroom on physical robots would be limited by reset noise.
- Heuristic Proposals: The "structured perturbations" used (axis offsets) are simple heuristics. Future work could involve training "advantage heads" or proposal networks to generate even more effective candidate actions.
- Compute Trade-offs: While the fail-gate helps, test-time selection still adds latency. Optimizing the WAM's imagination speed remains a critical path for real-time robotics.
Summary updated