Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail

Karthik IyerarXiv 2026 · arXiv

TL;DR

Standard forecast accuracy is a dangerously misleading metric for evaluating world models intended for planning. This paper demonstrates that while a model may excel at predicting outcomes within a known policy, it often fails spectacularly on novel actions unless domain-specific constraints—like basic price rules in retail—are explicitly baked into the architecture.

What Problem Does This Paper Address?

World models are designed to answer "what if" questions: how will the environment respond to an action it has never seen before? In fields like retail, models are trained on historical data generated by human experts (merchants). Because these experts follow a policy—such as only running promotions during high-demand weeks—the training data is heavily confounded.

A model might achieve high accuracy simply by learning the merchant's policy rather than the underlying physics of the world. The core challenge is that forecast accuracy on "logged" data tells us almost nothing about a model's ability to extrapolate to rare or unseen actions (e.g., a 60% discount when the maximum historical cut was 40%).

Methods

The author proposes a three-part test for action-conditioned world models:

  1. Held-out Actions: Training models on standard plans but testing them on extreme or "off-pattern" actions (e.g., deeper price cuts, lasting price moves, or no plan at all) where the ground truth is known.
  2. Correlation Analysis: Checking if better forecast accuracy actually predicts better performance on these held-out actions.
  3. Structured Baselines: Comparing complex world models against simpler models (like log-linear regressions) given the same structural constraints to isolate what the model "learns" versus what the "structure" provides.

The study evaluates several architectures, including World Models (encoder-recurrent-readout), LightGBM, and Log-linear models. It also tests "remedies" like JEPA pretraining, latent consistency checks, and the inclusion of explicit price rules (e.g., sales cannot fall as a discount deepens).

Datasets

The research utilizes both synthetic and real-world datasets:

  • Generated Worlds: Three synthetic retail environments (Log-linear, Bent, and Kinked) where the exact response to any price plan is known.
  • Real-World Retail: Three public datasets: Breakfast at the Frat, The Complete Journey, and M5. These are used to score models on the rarest plans actually executed in history.
  • Dosing World: A second domain involving medical dosing, where doctors (like merchants) dose sicker patients more heavily, creating similar confounding effects.

Key Insights and Ideas

  • The Accuracy Trap: In scenarios where plans follow demand, the link between forecast accuracy and simulation accuracy can actually reverse (rank correlation down to -0.69). Choosing a model version based on standard validation error might lead you to the worst possible simulator for planning.
  • Architecture vs. Inductive Bias: Modern self-supervised techniques like JEPA pretraining or latent consistency checks did almost nothing to improve performance on unseen actions.
  • Rules are Force Multipliers: Explicitly building domain knowledge into the model—such as the rule that a deeper discount shouldn't decrease sales—was the single most effective remedy. In the "bent" world, these rules dropped the world model's error from 1.39 to 0.27.
  • The "Kinked" Advantage: While simple log-linear models win when the world matches their functional form, world models with rules prove more robust in "kinked" worlds where the true response curve is complex and unknown.
Figure 1: Forecast accuracy vs. error on new plans. Note how the correlation (b, c) can dip below zero as planning strength increases.

Key Results

  • OOD Performance: Without rules, every model failed to distinguish between a temporary price cut and a lasting regular-price move, often overestimating the gain of a price drop by over 6x.
  • Real-World Validation: On the Breakfast at the Frat data, adding price rules lowered the world model's error on deep, pure price cuts from 0.63 to 0.56.
  • Dosing Stress Test: The findings generalized to the medical domain. Adding the "drug level" (state) helped, but the model still only predicted a fraction of the true gain until explicit dose rules were applied.
Figure 3: Error on new plans for every model and remedy (0 means the predicted change in sales is right). Medians over the 12 worlds of each shape. Filled: trained on the merchants’ history; open: with 5% random test plans. Rows after the first in each block, and the split rows, were added after the first results.

Limitations and Future Work

The primary limitation is that the "ground truth" for unseen actions is largely derived from generated worlds. While real-world data confirms some findings (like the benefit of rules for deep cuts), real retail environments contain complexities not captured here, such as stock-outs, competitor moves, and changing consumer sentiment.

The author suggests that future work should apply this "held-out action" framework to other critical world-model domains, such as robotics or autonomous driving, where simulators or logged interventions can provide the necessary "known answers" for testing trust.

Summary updated

Abstract

A world model is trusted to answer "what if": how its world responds to an action it has never seen. We give a test of that trust for any action-conditioned world model: actions held out of training by design and scored against a known answer; whether forecast accuracy predicts accuracy on those actions; and the same structure given to simpler models. We run it in retail, where the action is a price and promotion plan, across 2,853 fitted models. First, forecast accuracy is a fair guide between kinds of model (rank correlation with error on new plans 0.32 to 0.66) and a poor guide between versions of one world model: where merchants plan into high-demand weeks the link reverses, down to -0.69. Second, training the world model to learn the effect of actions (JEPA pretraining, a latent consistency check, adaptation at the plan date) leaves its error on new plans unchanged. Third, in generated worlds the recorded regular price and three price rules improve it. Given the same rules, a log-linear model is best when its curve can match the truth; in a world whose curve lies outside the rules, the world model with the rules is ahead of the log-linear model with the rules in 36 of 40 worlds, and does as well without its recurrent cell. On real data (Breakfast at the Frat, The Complete Journey and M5) the rules lower the world model's error on pure price cuts deeper than any it saw (0.63 to 0.56 on the Frat; 0.45 to 0.44, within chance, on The Complete Journey), but not on lasting price moves. Fourth, a small share of random test plans helps every model a little. In a generated dosing world, a stress test in a second domain, the first finding repeats, and the built-up drug level lowers the world model's error (2.86 to 1.91) though it still predicts only 0.12 to 0.32 of the true gain. The test carries to any domain where a world model is asked about actions outside its history.
BibTeX
@misc{iyer2026testing,
  author = {Karthik Iyer},
  title = {{Testing World Models on Actions They Have Never Seen: Held-Out Plans, Forecast Accuracy and Price Rules in Retail}},
  year = {2026},
  eprint = {2611.22537},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22537},
}
Figure