Do Synthetic-Market Metrics Predict Usefulness When Regimes Shift?

Video researchers learned the hard way that a generator can score well on standard metrics while producing output that humans immediately reject. The standard distance scores miss flicker, frame reordering and implausible motion, because those errors touch few pixels or disappear when frames are averaged over time. This note argues that generators of synthetic financial data, meaning models that produce artificial price histories for stress testing and training trading models, have the same problem and have not yet measured it. Published market generators report near-perfect scores on summary checks such as fat tails, volatility contrast between calm and crisis periods, and price-logic rules. Yet one of them passes only 1 of 56 tests of return autocorrelation, which is how much today's return depends on yesterday's. In finance the equivalent of the human viewer is the downstream user. Nobody has asked whether a generator that scores well on these checks actually gives a risk model or a strategy the right answer when the market moves into a regime it has not seen before.

We propose to build that test and a metric designed to pass it. The test borrows the validation method of video quality research. We assemble a population of generators, some of them deliberately corrupted. We then measure how well each candidate metric's ranking of these generators agrees with a ground-truth "usefulness" score. That score is computed on held-out crisis periods and plays the role human opinion scores play for video. The metric borrows the key structural idea of video evaluation: score marginal realism ("does each window look right") separately from temporal realism ("does it move right"). Its main change is that the temporal reference is conditioned on market regime, rather than the single fixed spectrum that video metrics assume.

Core Claim

Current evaluations of synthetic market data are blind to time in the same way Fréchet Video Distance is. They check aggregate statistics that can be preserved while the dependence structure over time is destroyed. Our claim is testable. First, across a population of generators, existing check batteries will correlate poorly with downstream usefulness in unseen crisis regimes. Second, a regime-conditioned spectral metric will correlate substantially better. Third, the gain will come specifically from the regime conditioning and the separate temporal axis, and ablations will isolate both. The direction on perceptual misalignment supplies the diagnosis and the validation method. The direction on market non-stationarity supplies the application setting and the reason a single stationary reference fails.

What Exists and What Is Missing

Two papers motivate the gap. Behavioral Proxy Conditioning for Financial Stress Scenario Generation with a Pretrained Diffusion Model adapts a pretrained time-series diffusion model to generate crises. It conditions generation on regime probabilities from a hidden Markov model (a model that infers unobserved market states from observed returns), on cross-market correlation and on volatility rank. It reports passing 27 of 28 regime-separation checks, stress scenarios 2.47 times more volatile than calm ones, 0% exact replay, a maximum drawdown of -14.98% and a Calmar ratio of 0.60 in a downstream portfolio test. It also reports passing only 1 of 56 return-autocorrelation checks. That is direct evidence that a battery of aggregate checks can be almost fully satisfied while temporal structure is wrong. Market-GAN labels price series as bear, flat or bull and generates data conditioned on those labels. It reports zero violations of point-in-time "Market Facts" such as the daily high never falling below the open, and it evaluates usefulness through forecasting error on downstream LSTM and GRU models. Neither paper asks whether its evaluation ranks generators the way downstream users would, least of all under regime shift. That question is what we add.

Two video papers supply the method. STREAM shows that Fréchet Video Distance barely reacts to swapped frames. It fixes this by taking a Fourier transform along time of per-frame embeddings, then scoring appearance from the zero-frequency component and motion from the rest of the spectrum against a single 1/f power law. We reuse the idea of separating the two axes. We reject the single-law assumption, because market return spectra differ sharply between calm and crisis periods, and that difference is the non-stationarity problem itself. Artifact-Aware Evaluation for High-Quality Video Generation argues that one scalar score hides diagnostic information and splits errors into appearance, motion and camera axes. We keep that multi-axis reporting. The stress-test design, which applies controlled corruptions and checks whether a metric reacts, comes from STREAM's frame-swap experiments.

Two regime models constrain the design. HireVAE and the Feature Saliency HMM both fix the number of regimes in advance. The HMM paper reports that performance degrades beyond two states, and HireVAE needs a special stabilization step to stop regime labels from swapping between epochs. We therefore keep the regime labeler deliberately simple, a two- or three-state HMM fit only on training data. We also test whether our conclusions survive changing it.

Research Program

The first component is the ground truth, which we call regime-shifted utility. For each generator, we train it only on data before a cutoff and then generate conditional on the crisis regime. We run three fixed downstream tasks on the synthetic output: forecasting next-period volatility, backtesting 1% Value-at-Risk, and estimating a fixed momentum strategy's drawdown distribution. We then compare each task's answer to the answer obtained on real held-out crisis windows, the 2020 and 2022 drawdowns. A generator's utility is the average rank of its error across the three tasks. This plays the role of human opinion scores.

The second component is the generator population. It includes real generators: GARCH, historical bootstrap, TimeGAN, an unconditional diffusion model, a Market-GAN reimplementation, and a proxy-conditioned diffusion model in the style of Dropout-FT. It also includes controlled corruptions of the real training data, analogous to frame swaps. Shuffling returns within a window keeps the marginal distribution but removes volatility clustering. A small first-order autocorrelation can be injected. Calm-period and crisis-period volatility dynamics can be spliced together under the wrong label. Sampling each corruption at several strengths gives roughly thirty to forty systems with known, graded defects.

The third component is the metric. For regime $r$, let $S^{real}_r(f)$ and $S^{gen}_r(f)$ be the average power spectra at frequency $f$ of three series computed within windows labeled $r$: returns, absolute returns, and rolling cross-asset correlation. The temporal score is

$$ T = \sum_r w_r \sum_{f>0} \left| \log S^{gen}_r(f) - \log S^{real}_r(f) \right|, $$

where $w_r$ weights each regime equally, so that rare crisis windows are not swamped by calm ones. The marginal score is computed separately from within-window distributions using precision and recall, following STREAM. The two scores are always reported as a pair and never merged. Spurious autocorrelation raises the low-frequency power of returns. Missing volatility clustering flattens the spectrum of absolute returns. The metric is therefore built to catch exactly the failures the check batteries miss.

Baselines

The naive reference point is the check battery itself: the fraction of stylized-fact and regime-separation checks passed, as in the behavioral-proxy paper, plus the Market Facts violation count from Market-GAN. This is the evaluation the field reports today. We reimplement it on identical synthetic samples. The strongest published figures here describe single generators, not rankings: 27 of 28 regime-separation checks passed and 1 of 56 autocorrelation checks passed, both from Behavioral Proxy Conditioning. No reported result was found on Lacuna for the correlation of any synthetic-market metric with downstream utility across generators, so every baseline correlation will be computed by us.

The strongest existing competitor is a discriminative score. We train a classifier to tell real windows from synthetic ones and report its held-out accuracy. It is widely used and needs no hand-chosen statistics, so it could in principle detect any temporal flaw. For fairness, it sees the same windows as our metric, receives the same regime labels as an input feature, and gets a hyperparameter search matched to the tuning spent on our frequency binning and weights. The third baseline is a video-style temporal metric without regime conditioning: the same spectral score computed on all windows pooled, fit to a single reference spectrum as STREAM does. The fourth baseline is the downstream error of a forecasting model trained on synthetic data and tested on real data, in the style of Market-GAN, computed on calm validation data only. Its role is described in the next section.

Two ablations isolate the key ingredients. One removes the regime conditioning, and the other removes the marginal/temporal split by collapsing the two scores into one. Beating the baselines means achieving a higher Spearman rank correlation with regime-shifted utility across the generator population than the battery and the discriminator, with a bootstrap confidence interval over generators that excludes zero for the difference. The comparison uses a test for dependent correlations, because all metrics are scored on the same generators. We also require that the regime-conditioned version beat its pooled ablation. Otherwise the non-stationarity half of the proposal adds nothing.

Why not just…?

Why not just add standard autocorrelation tests to the check battery? This is the cheapest fix and deserves a direct test. Autocorrelation tests are pass/fail per lag and per series, so they flag spurious return autocorrelation. They do not rank how far off a generator is, and they say nothing about volatility clustering or regime-specific dynamics. The behavioral-proxy paper's 1/56 result shows these tests can flag a problem. It does not show that the pass count predicts downstream harm. We include "battery plus autocorrelation tests" as a named baseline. If it matches our metric's correlation with utility, we will report that the spectral machinery is unnecessary.

Why not just use a real-versus-synthetic classifier? This is the real threat. A flexible classifier could learn every temporal flaw our spectra encode. Two considerations push the other way. Classifier accuracy saturates near 100% for most imperfect generators, which compresses rankings. It also gives no diagnostic axis, the complaint Artifact-Aware Evaluation makes against single scores. The experiment settles the question. If the classifier's correlation with utility equals or exceeds ours, the contribution becomes the benchmark and the corruption suite, and we will say so.

Why not just evaluate every generator by downstream utility directly? Downstream utility is the ground truth here, but it cannot serve as a routine metric, because it needs real crisis data that practitioners do not have in advance. The practical question is whether something computed on data available before the crisis predicts behavior during it. The calm-period forecasting baseline tests exactly that version of the idea. If calm-period utility already predicts crisis-period utility, regime conditioning is unnecessary, and the experiment will show it.

Proposed Experiments

The main data are daily OHLCV series for the seven global equity indices used by the behavioral-proxy paper (2000–2024, 36-day windows). A replication uses the 29 Dow Jones stocks with bear/flat/bull labels in Market-GAN's Contextual Market Dataset. The pre-cutoff period is used for training generators and fitting regimes. The 2020 and 2022 stress windows are used only to compute utility.

The first decisive check is the corruption sensitivity test, the market analog of STREAM's frame swaps. The battery should score the volatility-clustering shuffle at almost the same value as real data at every corruption strength. Our temporal score should rise monotonically with corruption strength, while the marginal score stays flat. The second check is the main alignment result: Spearman correlation with regime-shifted utility across the full generator population, for every baseline and ablation. The third check tests robustness to the regime labeler. It swaps the HMM for Market-GAN's trend-based labels and varies the number of states between two and three. A metric whose advantage disappears under a different labeler is measuring the labeler, not market realism. The fourth check tests cross-market transfer. It tunes all metric choices on the index data and reports alignment unchanged on the Dow stocks, since AlphaQCM documents that findings from one market often weaken in another. A positive result on all four checks would give synthetic-market evaluation what video evaluation gained from STREAM: evidence that the standard numbers were not measuring what practitioners need.