TL;DR
Standard forecast accuracy is a dangerously misleading metric for evaluating world models intended for planning. This paper demonstrates that while a model may excel at predicting outcomes within a known policy, it often fails spectacularly on novel actions unless domain-specific constraints—like basic price rules in retail—are explicitly baked into the architecture.
What Problem Does This Paper Address?
World models are designed to answer "what if" questions: how will the environment respond to an action it has never seen before? In fields like retail, models are trained on historical data generated by human experts (merchants). Because these experts follow a policy—such as only running promotions during high-demand weeks—the training data is heavily confounded.
A model might achieve high accuracy simply by learning the merchant's policy rather than the underlying physics of the world. The core challenge is that forecast accuracy on "logged" data tells us almost nothing about a model's ability to extrapolate to rare or unseen actions (e.g., a 60% discount when the maximum historical cut was 40%).
Methods
The author proposes a three-part test for action-conditioned world models:
- Held-out Actions: Training models on standard plans but testing them on extreme or "off-pattern" actions (e.g., deeper price cuts, lasting price moves, or no plan at all) where the ground truth is known.
- Correlation Analysis: Checking if better forecast accuracy actually predicts better performance on these held-out actions.
- Structured Baselines: Comparing complex world models against simpler models (like log-linear regressions) given the same structural constraints to isolate what the model "learns" versus what the "structure" provides.
The study evaluates several architectures, including World Models (encoder-recurrent-readout), LightGBM, and Log-linear models. It also tests "remedies" like JEPA pretraining, latent consistency checks, and the inclusion of explicit price rules (e.g., sales cannot fall as a discount deepens).
Datasets
The research utilizes both synthetic and real-world datasets:
- Generated Worlds: Three synthetic retail environments (Log-linear, Bent, and Kinked) where the exact response to any price plan is known.
- Real-World Retail: Three public datasets: Breakfast at the Frat, The Complete Journey, and M5. These are used to score models on the rarest plans actually executed in history.
- Dosing World: A second domain involving medical dosing, where doctors (like merchants) dose sicker patients more heavily, creating similar confounding effects.
Key Insights and Ideas
- The Accuracy Trap: In scenarios where plans follow demand, the link between forecast accuracy and simulation accuracy can actually reverse (rank correlation down to -0.69). Choosing a model version based on standard validation error might lead you to the worst possible simulator for planning.
- Architecture vs. Inductive Bias: Modern self-supervised techniques like JEPA pretraining or latent consistency checks did almost nothing to improve performance on unseen actions.
- Rules are Force Multipliers: Explicitly building domain knowledge into the model—such as the rule that a deeper discount shouldn't decrease sales—was the single most effective remedy. In the "bent" world, these rules dropped the world model's error from 1.39 to 0.27.
- The "Kinked" Advantage: While simple log-linear models win when the world matches their functional form, world models with rules prove more robust in "kinked" worlds where the true response curve is complex and unknown.
Key Results
- OOD Performance: Without rules, every model failed to distinguish between a temporary price cut and a lasting regular-price move, often overestimating the gain of a price drop by over 6x.
- Real-World Validation: On the Breakfast at the Frat data, adding price rules lowered the world model's error on deep, pure price cuts from 0.63 to 0.56.
- Dosing Stress Test: The findings generalized to the medical domain. Adding the "drug level" (state) helped, but the model still only predicted a fraction of the true gain until explicit dose rules were applied.
Limitations and Future Work
The primary limitation is that the "ground truth" for unseen actions is largely derived from generated worlds. While real-world data confirms some findings (like the benefit of rules for deep cuts), real retail environments contain complexities not captured here, such as stock-outs, competitor moves, and changing consumer sentiment.
The author suggests that future work should apply this "held-out action" framework to other critical world-model domains, such as robotics or autonomous driving, where simulators or logged interventions can provide the necessary "known answers" for testing trust.
Summary updated