Decision Exchange Rates for Synthetic Demand in Multi-Product Inventory

Retailers increasingly train ordering policies on simulated demand: real logs are scarce for new products and stores, a single training run can take a day, and real sales are censored whenever shelves are empty. Everyone who relies on these simulators faces the same unanswered question: how much is a synthetic demand trajectory worth compared with a real one, for the decision that actually matters? We propose to measure a decision exchange rate, defined as the number of synthetic trajectories needed to buy the same reduction in real operating cost as one real trajectory. We measure it in a multi-product setting where items compete for shared capacity and customers switch to a substitute when their first choice is out of stock.

Our central hypothesis is that, in this setting, the value of synthetic data depends mainly on whether the generator reproduces dependence across products. That means joint demand spikes, shared shocks, and spillover of demand from a stocked-out item to its substitutes. Per-series realism matters much less, and per-series realism is what current synthetic time-series metrics mostly reward. If the hypothesis holds, the field's default validation checks will approve generators that produce poor inventory policies. We will also test a cheap diagnostic that should catch this failure: whether synthetic and real data rank a fixed panel of policies in the same order.

Core Claim and What Is New

Synthetic data evaluation already distinguishes fidelity, diversity, utility and privacy. The survey by Evaluation is key catalogs these metrics and names "train on synthetic, test on real" (TSTR) as the reference utility test. It also reports that practitioners mostly reuse ad-hoc metrics from generator papers. The only quantitative exchange rate in the file comes from multi-object tracking. There, On the Equivalency, Substitutability, and Flexibility of Synthetic Data finds that roughly 30 synthetic frames are worth one real frame and that 60% of real data can be replaced without loss. It also finds that synthesizing complex interactions closed the domain gap more than visual realism did. That last result motivates our hypothesis. Its inventory analogue is that interactions between products, not single-series realism, should govern synthetic value.

On the inventory side, several strong learned policies are trained against simulators, but none measures how good those simulators are. The direct-backpropagation simulator in Deep Inventory Management was validated only through a live A/B test. The offline RL work in Offline Dynamic Inventory and Pricing Strategy treats its synthetic economy as ground truth. Both are single-product, and so is DeepStock. Business Metric-Aware Forecasting for Inventory Management shows the "accuracy paradox": the forecast with the lowest squared error can produce worse inventory costs. That is direct evidence that statistical closeness and decision value can diverge, but the paper concerns forecasters, not generators. Our work differs from all of these in one way. We make the generator the object being evaluated, and we score it in units of real inventory cost on a coupled, multi-product system.

We define the exchange rate as follows. Let $C(n_r, n_s)$ be the real-evaluated total cost (holding, lost-sales penalty and order-variance cost, following the cost objective of Business Metric-Aware Forecasting). It is the cost of a policy trained on $n_r$ real and $n_s$ synthetic trajectories. The exchange rate at real budget $n_r$ is

$$\rho(n_r) = \frac{n_s}{\Delta}\quad \text{where } \Delta \text{ solves } C(n_r-\Delta,\, n_s) = C(n_r,\, 0),$$

so $\rho$ counts how many synthetic trajectories replace one real trajectory without raising real cost. A generator with $\rho = \infty$ is worthless. One with small $\rho$ is a strong substitute.

Research Program

The environment is a periodic-review system with lost sales. Each group of 3–8 substitutable items is drawn from one subcategory at one store and shares a capacity limit. Replenishment has a positive lead time. When an item stocks out, a fraction of its unmet demand moves to in-stock substitutes. This is the coupling that, as the file notes, breaks the single-product counterfactual "order a lot, then evaluate smaller orders" trick of Learning to Order.

The policy learner is held fixed across all conditions so that only the training data varies. It is a shared neural ordering policy trained by backpropagating through the differentiable simulator, as in Deep Inventory Management. Its output is structured as a base-stock target minus inventory position, following the regularization in DeepStock, and orders are projected onto the capacity limit.

Measuring "test on real" is itself hard here, because real logs record sales, not demand. We therefore evaluate in two tiers.

The first tier is a known-reality setting. We calibrate a rich ground-truth simulator to Corporación Favorita: per-item negative-binomial demand with store-level shared shocks, promotion effects, and an explicit substitution matrix. Generators see only censored sales that this simulator produces under a logging policy. Because true demand is known, $C$ can be computed exactly, and we can vary substitution strength and sparsity on purpose.

The second tier is real replay on Favorita and M5. We restrict evaluation to high-velocity items and periods where stockouts are rare, so that sales are a reasonable proxy for demand, and we state openly that this introduces bias.

We will evaluate four generator families. All are fit on the same censored training data. The first is independent per-SKU negative-binomial sampling from the top-down hierarchical model of Scalable Probabilistic Forecasting. The second is a block bootstrap of real store-days, which preserves cross-product dependence by construction. The third is a deep joint generator, a least-squares GAN over item-by-day matrices, borrowed from the scenario generator of Fed-LSGAN without the federated part. The fourth is the same deep generator with demand uncensored before fitting, using Kaplan–Meier imputation as in Offline Dynamic Inventory and Pricing.

For every generator we compute $\rho$, which is the expensive ground truth. Alongside it we compute the cheap metrics. These are a classifier two-sample test, embedding-based distance, TSTR forecast error (weighted absolute percentage error, WAPE), spectral predictability from Time Series Forecastability Measures, and our proposed diagnostic. The diagnostic takes about twenty base-stock and $(s,S)$ policies, evaluates each on synthetic and on real data, and reports the Kendall rank correlation between the two cost orderings. We also report an explicit cross-product statistic: the joint exceedance rate of the 95th percentile within a substitution group.

Baselines

The naive reference point is training on real data only. This is the curve $C(n_r, 0)$ itself, and every exchange rate is defined against it. We trace it at real budgets from 2% to 100% of available groups so that the low-data regime, where synthetic data should matter most, is visible.

The strongest simple generator is the block bootstrap of real history. It captures cross-product dependence exactly but cannot extrapolate, and it inherits the censoring of the logs. We expect it to be the hardest baseline to beat at moderate real budgets.

The strongest existing modeling component is the negative-binomial hierarchical model of Scalable Probabilistic Forecasting. It is cheap and accurate on marginals; the paper reports a top-50 finish out of nearly 900 teams in the M5 uncertainty track. Sampled independently per item, it is the natural "marginals right, coupling wrong" competitor.

For the policy learner itself, the closest reported figure on Favorita is a 54.0% total-cost improvement of a business-aware LSTM over its squared-error-trained version under heavily imbalanced costs, from Business Metric-Aware Forecasting. We will reproduce that single-product setting as a sanity check on our cost implementation before adding coupling. No reported result was found on Lacuna for synthetic-to-real exchange rates in any inventory or supply-chain task. The only reported exchange rate is the 30:1 tracking figure, which we cite as context, not as a target.

The key ablation isolates cross-product dependence. We take the best deep generator and destroy dependence between items within each synthetic sample by independently permuting items across samples. This leaves every marginal series, and therefore every per-series metric, unchanged. If $\rho$ worsens sharply under this permutation, dependence is the driver. A second ablation removes uncensoring, to isolate censoring bias.

For fairness, every condition uses the same policy architecture, the same simulator, the same number of gradient steps, and the same hyperparameter budget of 20 trials tuned on a real validation split. All generators are fit on identical censored training data. Beating a baseline means a lower $\rho$, with bootstrap confidence intervals over substitution groups that do not overlap, at real budgets of 10% or less. For the diagnostic, beating the alternatives means a higher Spearman correlation with $\rho$ across generators than the classifier test, embedding distance, or TSTR WAPE achieve.

Why not just…?

Why not just use TSTR with forecast accuracy, training a forecaster on synthetic data and scoring WAPE on real? The accuracy paradox in Business Metric-Aware Forecasting shows that squared-error quality and cost can rank models in opposite orders. More importantly, a per-item forecaster cannot be hurt by missing substitution, because it never sees the coupling. Our experiment reports the correlation between TSTR WAPE and $\rho$ directly. If that correlation is as high as our diagnostic's, the simpler test wins and we will say so.

Why not just bootstrap real history instead of building generators? This is a real threat. At large real budgets, block bootstrap may well match or beat every learned generator, because it is reality resampled. Its limits are structural. It cannot produce demand for new items or stores, and it replays censored sales, so it under-represents exactly the stockout-driven spillover the policy must learn. We test both limits. A cold-start split evaluates on substitution groups absent from training, and the tier-one simulator lets us vary censoring severity. If bootstrap dominates even there, the honest conclusion is that learned generators are not yet worth using for this task.

Why not just validate generators with a classifier two-sample test or embedding distance? The survey in Evaluation is key notes that embedding-based scores inherit the blind spots of their backbone. Time Series Forecastability Measures shows that dynamical measures become unstable once more than about 80% of observations are zero, which is common at the SKU level. The permutation ablation is the decisive check. It leaves per-series distributions intact, so per-series scores should be unchanged, while we predict $\rho$ degrades. A classifier given whole item-group matrices could in principle detect the permutation, so we run it both per series and per group and report both.

Why not just collect more real data? For established items this is often the right answer, and the real-only curve shows where synthetic data stops helping. New products, new stores, and settings where the logging policy rarely stocked out offer no real data that reveals spillover. That is the regime the cold-start and censoring experiments target.

Decisive Checks

The proposal succeeds if three things hold. First, at real budgets of 10% or less, at least one generator achieves a finite, stable $\rho$. Second, the permutation ablation raises $\rho$ substantially while leaving per-series metrics within noise. Third, the policy-ranking diagnostic correlates with $\rho$ across generators better than the classifier test, embedding distance, and TSTR WAPE do. It fails informatively if $\rho$ is insensitive to the permutation, which would mean coupling does not matter at realistic substitution rates. It also fails informatively if bootstrap dominates everywhere, which would mean learned generators add nothing here. Either outcome tells practitioners something they currently cannot know before deployment.