TL;DR
Being a great forecaster doesn't make you a good "world modeler." This paper demonstrates that state-of-the-art temporal foundation models—even those designed for causal inference—fail to accurately predict the effects of price changes when unobserved factors influence both prices and sales. Surprisingly, providing more data to these models often improves their forecasts while leaving their causal predictions stagnant or even making them worse.
What Problem Does This Paper Address?
In retail, forecasting sales and setting prices are two different beasts. A forecast asks: "Given what usually happens, how many units will I sell next week?" A pricing decision asks: "If I change the price by 10%, what will happen to my sales?"
The problem is confounding. Retailers often discount products when they anticipate high demand (e.g., during a holiday or a local event). If a model isn't aware of that anticipated demand, it sees "low price + high sales" and incorrectly attributes all those sales to the discount. This is a classic "correlation is not causation" trap. The authors investigate whether modern foundation models (FMs)—which are trained on massive amounts of diverse data—can "see through" this confounding to recover the true causal response.
Methods
The authors evaluate a suite of models across three categories:
- Tabular Foundation Models: TabPFN, TabICLv2, and TabFM.
- Time-Series Foundation Models: Chronos-2.
- Causal Foundation Models: CCPFN (specifically pretrained for causal effects).
- Baselines: LightGBM (standard ML) and Double Machine Learning (an econometric method designed to handle confounding using instrumental variables).
The evaluation focuses on two tasks:
- Forecasting: Predicting sales at observed prices.
- Counterfactual Prediction: Predicting the ratio of sales change after a hypothetical 10% price intervention.
Datasets
- CausalDemand: A newly introduced benchmark that simulates retail sales calibrated to real-world IRI store records. Crucially, it provides two versions of the same data: one unconfounded (prices are set randomly) and one confounded (prices respond to unobserved demand shocks). This allows the authors to know the "ground truth" of an intervention.
- Dominick’s Finer Foods: A real-world dataset of weekly scanner data used to verify if models at least get the direction of price effects right (i.e., do sales go down when prices go up?).
Key Insights and Ideas
- The Accuracy Paradox: High forecast accuracy is a poor proxy for causal accuracy. A model can be incredibly precise at predicting sales while being completely wrong about price elasticity.
- More Data $\neq$ Better Causal Insights: In confounded settings, giving a model more "context" (in-context examples or training data) helps it learn the correlation better, which actually entrenches its causal errors. As shown in the figures below, forecast error drops with more context, but counterfactual error remains stubbornly flat or even increases.
- The "Causal" Label Isn't Enough: Even CCPFN, a model pretrained for causality, struggled. The authors argue this is because these models are often trained under the "ignorability" assumption (that all confounders are observed), which is almost always violated in real-world retail data.
- Instrumental Variables (IV) are a Double-Edged Sword: In econometrics, IVs are the standard fix for confounding. However, simply feeding "raw" instruments into a foundation model actually increased error. To work, the instruments had to be supplied as "residuals"—the part of the price that the instruments cannot explain.
Key Results
- Directional Errors: On real retail data, the most accurate forecasters often failed the "law of demand." While Double ML (using instruments) predicted sales would fall after a price increase 100% of the time, models like Chronos-2 and TabPFN predicted the wrong direction on 6-21% of the data.
- Counterfactual Error: In the confounded simulation, most foundation models had a counterfactual error near 0.08–0.10 WMAPE. For context, a model that simply predicts "no change" would have an error of 0.144.
- IV Performance: When supplied with raw instruments, the counterfactual error for tabular models jumped significantly (e.g., TabPFN went from 0.094 to 0.193), performing worse than a "no change" baseline. Only when supplied as residuals did the error drop (to ~0.05).
Limitations and Future Work
The study highlights that current foundation models are primarily "associative" rather than "causal." One major limitation is that these models were evaluated in their "off-the-shelf" state; fine-tuning on causal objectives might yield different results.
The authors suggest that for foundation models to truly act as "world models," we need new evaluation benchmarks that prioritize response accuracy over simple forecast accuracy. Future work should explore how to bake econometric principles, like instrumental variable logic, directly into the architecture or prompting strategies of temporal foundation models.
Summary updated