Generative models for time series, such as TimeGAN, TimeVAE and DiffusionTS, are increasingly used to stand in for scarce or private real data. They are usually judged by whether their samples look statistically like real ones, or by whether a small neural network trained afterward can tell real from fake. Neither check asks the question a dynamicist would ask: when you replay a synthetic series as a trajectory, does it trace out the same shape in the space of system states? In that space (the phase space), a sustained oscillation is a loop, a system with two stable operating points is two separate clusters, and chaos is a folded, recurring set. A generator could match every frequency and marginal statistic of the real data and still turn a loop into an inward spiral, or merge two basins into one. A model trained on that output would then learn the wrong long-term behavior.
This proposal builds a stress test and an audit for that failure. The audit rebuilds each series as a point cloud in phase space by stacking delayed copies of the signal, a standard reconstruction known as delay embedding. It then measures the shape of that cloud with persistent homology, a tool from topological data analysis that counts connected pieces and loops and records how robust each one is across scales. The real and synthetic collections are compared with optimal transport. The main contribution is not another scalar score. It is a controlled test of whether nonlinear phase-space shape determines how useful synthetic data is for downstream training, beyond what linear and short-window statistics already capture. The test uses counterfactual synthetic datasets that match real data on those statistics but break its topology on purpose.
Core Claim
The closest existing work is DMD-GEN (Abbahaddou & Aboussalah, 2025). It fits a linear operator $x_{t+1}\approx Ax_t$ to each real and synthetic series, treats the leading modes as points on a Grassmann manifold (the space of linear subspaces), and compares real and synthetic by optimal transport. It is training-free, and under controlled mode collapse it responds monotonically where discriminative and predictive scores fluctuate. It is also linear by construction. A stable limit cycle and a slowly decaying spiral with the same frequency differ only by a small shift of an eigenvalue. A bistable system and a single-well system with a matching spectrum can look almost identical to it.
The tools to see these differences already exist, but only for classification. Persistent homology of attractors for action recognition adds edges between consecutive time points before computing persistence ("temporal simplicial complexes"). With this, it separates motion-capture actions at 96.48% mean accuracy, against 52.44% for classical chaos invariants. Shape Distributions of Nonlinear Dynamical Systems for Video-Based Inference. reaches similar accuracy with cheaper histograms of local phase-space shape. Neither paper has been used to judge a generator.
Our claim has two parts. First, on a suite of synthetic datasets that match the real data on marginals, linear dynamic modes and short-window statistics while altering attractor topology, a persistence-based distance detects the alteration where DMD-GEN and discriminative scores do not. Second, and more important, the same alteration measurably hurts train-on-synthetic, test-on-real performance on tasks that depend on long-horizon behavior. If the second part fails, then topology is diagnosable but irrelevant to reliance on synthetic data. That would be a useful negative result for practitioners.
Research Program
The audit is defined as follows. Fix an embedding dimension $m$ and delay $\tau$ on the real data only, using false-nearest-neighbor and mutual-information heuristics, and reuse them for every generator. For each sequence $s$, build the delay-embedded cloud and its temporal complex, and compute persistence diagrams $D_0(s)$ (connected pieces) and $D_1(s)$ (loops). Write $P_R$ and $P_G$ for the empirical collections of diagrams from real and generated sequences. The audit distance is
$$ \mathcal{T}(P_R,P_G)=\min_{\pi\in\Pi(P_R,P_G)}\ \mathbb{E}_{(r,g)\sim\pi}\big[W_1(D_0(r),D_0(g))+W_1(D_1(r),D_1(g))\big], $$
where $\Pi$ is the set of couplings between the two collections and $W_1$ is the 1-Wasserstein distance between diagrams. The optimal-transport outer layer is taken directly from DMD-GEN. Only the ground cost changes, from principal angles between linear subspaces to distances between topological summaries. Because the comparison is set-to-set, the audit can detect a missing basin (mode collapse) as well as a distorted attractor.
Two practical choices matter. Computing diagram distances is expensive, which the persistent-homology paper itself flags. We therefore subsample sequences and landmark points, and we report cost against DMD-GEN. For multichannel data such as the 28-feature Energy set, delay embedding the full vector explodes in dimension. That is the same scaling wall Phase2vec reports for its grid-based encoder. We embed the leading principal components and a few individual channels separately, and we treat the high-dimensional case as a stated limitation rather than hide it.
The stress suite is the part a benchmark cannot provide. It uses three dynamical systems with known phase portraits: Van der Pol (a single limit cycle), a forced or unforced bistable Duffing oscillator (two basins), and Lorenz (a chaotic attractor). Sampled trajectories with observation noise play the role of "real" data. For each system we build counterfactual synthetic sets.
The first counterfactual set uses spectrum-preserving surrogates. These randomize the Fourier phases of each series, which keeps the power spectrum and linear autocorrelation exactly and destroys nonlinear structure. The second collapses the limit cycle into a slowly damped spiral. The damping rate is tuned so that DMD eigenvalues move by less than the gap separating real-versus-real bootstrap resamples. The third drops one Duffing basin while reweighting samples so that marginals stay close. The fourth splices together short windows from the true system in shuffled order, which preserves short-window statistics and breaks global recurrence.
We also train TimeGAN, TimeVAE and DiffusionTS on each system across seeds and training stages. This yields a population of real generator checkpoints whose topological fidelity varies naturally, not by design.
Every synthetic set is scored downstream by training on synthetic data and testing on real. There are two tasks. One is multi-step forecasting at horizons long enough that attractor shape matters. The other is regime prediction: which basin a trajectory ends in, or whether an oscillation persists. The real benchmarks from DMD-GEN (Sines, Stock, Energy, ETTh) are then audited in the same way, and the tasks are forecasting and next-window prediction.
Baselines
The naive reference point is a marginal and short-window statistical distance: maximum mean discrepancy with an RBF kernel on raw values, and a separate version on short delay windows. It is computed on the same samples. It counts as beaten if the audit detects counterfactuals that this distance ranks as indistinguishable from real-versus-real resamples.
The strongest existing method is DMD-GEN (Abbahaddou & Aboussalah, 2025). It is the only training-free, geometry-based generator metric in the file, and it already beats black-box scores on monotonicity under collapse. The file reports qualitative findings only: TimeGAN scored best on ETTh, Stock and Energy, and DiffusionTS best on Sines, across DMD-GEN, Discriminative and Predictive scores. There is no reported result found on Lacuna for DMD-GEN's per-cell numerical values. We will rerun it with the same number of modes as the paper, the same batches and the same optimal-transport solver as our audit, and give it a matched hyperparameter search over the number of modes. Beating it means two things. The audit must separate the damped-spiral and dropped-basin counterfactuals from real data at a bootstrap significance level where DMD-GEN does not. And across the checkpoint population, the audit's rank correlation with downstream error must be higher than DMD-GEN's, with confidence intervals from resampled checkpoints that do not overlap.
The black-box baselines are Discriminative Score and Predictive Score, run with the standard post-hoc recurrent networks and a fixed training budget of five seeds each. The seed variance is reported, since instability is part of the comparison.
Three ablations isolate the key ingredients. The first computes persistence on the raw signal without delay embedding, which tests whether reconstruction matters. The second uses a standard Vietoris–Rips complex without temporal edges; that variant scored 93.68% versus 96.48% for the temporal complex in the classification setting. The third replaces persistence with the DT2 shape histograms, which asks whether topology adds anything over cheaper local geometry.
Why not just…?
Why not just compute Lyapunov exponents or correlation dimension on real and synthetic data and compare them? On the motion-capture benchmark, classical chaos invariants reached only 52.44% accuracy where temporal persistence reached 96.48% (Persistent homology of attractors for action recognition). They are also known to be unstable on short, noisy series, which is exactly what generators emit in batches. They additionally describe nothing for non-chaotic failures such as a missing basin. We will still include them as a baseline in the stress suite so the comparison is empirical rather than inherited.
Why not just use DMD-GEN, which is already geometric and cheap? This is the most serious competitor. It may well win on the real benchmarks if their failures are mostly spectral. The stress suite is designed to settle the question. The spiral and surrogate counterfactuals are built to sit inside DMD-GEN's noise floor, so detecting them is a clean test of nonlinear sensitivity. The checkpoint-population correlation then shows whether that sensitivity matters for downstream use. If DMD-GEN correlates with downstream error as well as the audit does on naturally trained generators, we will report that topology adds diagnostics but not predictive value.
Why not just train a discriminator, or compute MMD on delay-embedded windows? Discriminators are costly and unstable, and they do not respond monotonically to collapse (Abbahaddou & Aboussalah, 2025). Raw time-series classifiers like TapNet scored F1 below 0.3 on stability recognition where a phase-space encoder exceeded 0.9 (Phase2vec). MMD on short delay windows is a real threat because it removes the permutation-invariance objection to marginal metrics. The window-splicing counterfactual targets it directly: local windows are exact, and only global recurrence is broken. If windowed MMD catches that counterfactual and tracks downstream error as well as the audit, the audit reduces to an interpretability tool, and we will say so.
Proposed Experiments
The first decisive check is detection. For each of the three systems and each counterfactual, we compute every metric's distance divided by the spread of its real-versus-real bootstrap distances. A counterfactual counts as detected when this ratio exceeds the 95th percentile of the null distribution. The prediction is that the audit detects all four counterfactual families. DMD-GEN should miss the surrogates and, by construction, the spiral. Windowed MMD should miss the spliced recurrence.
The second decisive check is relevance. Across roughly sixty generator checkpoints per system, we compute Spearman correlation between each metric and error on the real data after training on synthetic data, separately for long-horizon forecasting and regime prediction. We also fit a regression of downstream error on DMD-GEN plus windowed MMD, then add the audit, and report the increase in explained variance. A negligible increase falsifies the central claim.
The third check transfers the audit to Sines, Stock, Energy and ETTh with the same three generators. It reports whether the audit's ranking agrees with the qualitative ranking in the file (TimeGAN best on the real datasets, DiffusionTS best on Sines). It also reports which diagram features, lost loops or merged pieces, account for any disagreement.
Finally, we report wall-clock cost against DMD-GEN at matched batch sizes, so readers can judge whether any gain in detection justifies the extra computation.