Auditing Synthetic Data by Whether Hidden Subpopulations Can Still Be Recovered

Synthetic data is often judged by whether it looks like the real data. Common checks compare one column at a time, compare pairs of columns, or train a classifier to tell real rows from synthetic ones. Many real datasets, though, are blends of several hidden subpopulations, each following its own rule linking inputs to outcomes. Examples are customers who respond to price in opposite ways, or listeners who move between genres in different patterns. A synthetic generator can reproduce every column and every pairwise correlation and still merge those rules into one average rule. Anyone who then uses the synthetic data to study heterogeneity, fit per-group models, or test a personalization method gets a wrong answer and has no warning.

We propose a direct audit for this failure. Take a mixture-recovery algorithm with known guarantees, meaning one that recovers the hidden component rules from unlabeled data. Run it on the real data and on the synthetic data, then compare what comes back: the component parameters, the mixing proportions, and the subspace those components span. If the generator kept the hidden structure, both runs should return the same components, up to the error expected from finite samples. If it did not, the audit says which component was lost or distorted. This is more specific than reporting that the data "differs."

Core Claim

Hidden mixture structure in the conditional relationship between inputs $x$ and an outcome $y$ is a fidelity property that existing synthetic-data checks are poorly placed to see. Methods from unlabeled mixture identification measure it cheaply and at small sample sizes.

The sharpest form of the claim comes from a construction that is standard in mixed linear regression. Suppose two components have equal weight and opposite parameter vectors, so that $y = x^\top w + \epsilon$ for half the rows and $y = -x^\top w + \epsilon$ for the other half. Then every first- and second-order statistic linking $x$ and $y$ is the same as for pure noise. The structure appears only in the response-weighted fourth-moment matrix $$M = \frac{1}{N}\sum_{i=1}^N y_i^2\, x_i x_i^\top,$$ whose top eigenvectors span the component vectors. This is the spectral initialization of Alternating Minimization for Mixed Linear Regression. Column-wise and pairwise checks therefore cannot detect a generator that loses this structure. The marginal and pairwise tiers of Structured Evaluation of Synthetic Tabular Data are blind by construction. Their leave-one-out conditional tier is not blind in principle, but it is not aimed at this structure either. That paper already shows that generators which pass low-order tiers degrade at higher-order tiers. We go further: we say which higher-order structure matters for heterogeneity questions and measure it directly.

The audit has two levels. The cheap level computes $M$ on real data ($M_R$) and on synthetic data ($M_S$). It then reports the largest principal angle between their top-$k$ eigenspaces, $\theta_k(M_R, M_S)$, together with a bootstrap null obtained by computing the same angle between two halves of the real data. This is a closed-form, discriminator-free statistic. The full level runs spectral initialization followed by alternating minimization on each dataset, which returns component estimates $\{\hat w_j, \hat\pi_j\}$, and reports the matched component error $$D = \min_{\sigma} \max_j \left( \|\hat w^R_j - \hat w^S_{\sigma(j)}\| + |\hat\pi^R_j - \hat\pi^S_{\sigma(j)}| \right),$$ where $\sigma$ ranges over permutations of the $k$ components. For sequential data, the same logic applies with the subspace-clustering pipeline of Learning Mixtures of Markov Chains and MDPs as the recovery engine and transition matrices as the components.

Relation to Prior Work

Two papers in the file connect mixtures and synthetic data, and neither does this. Synthetic Tabular Data Validation: A Divergence-Based Approach trains a probabilistic discriminator to estimate the real-to-synthetic density ratio and from it the joint divergence. It uses Gaussian mixtures only as test distributions for that discriminator and never recovers components. The same paper is also the main constraint on our design, since it reports that the discriminator needs large real and synthetic samples to avoid biased estimates. Structured Evaluation of Synthetic Tabular Data motivates the work: its PCC synthesizer, a model that clusters rows and columns into latent groups, and synthpop beat CTGAN and TVAE on small structured tables. That result hints that latent group structure matters. Its evaluation hierarchy, however, never asks whether specific latent components survive.

The mixture-identification papers supply the method. Alternating Minimization for Mixed Linear Regression gives the spectral start. Alternating Minimization Converges Super-Linearly for Mixed Linear Regression gives fast refinement, reporting 5–6 iterations against 45–48 for gradient descent. Iterative Least Trimmed Squares for Mixed Linear Regression gives a trimmed variant for the case where generators emit junk rows. These papers have used synthetic data only as a testbed with known ground truth. We reverse that role and use the recovery algorithm to judge the synthetic data.

Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality is the closest in spirit. It compares likelihood surfaces of real and simulated data for a four-parameter knowledge-tracing model using a brute-force grid. We replace the grid with polynomial-time spectral recovery and apply the idea to generators rather than to a scientific model.

Research Program

The work has three parts.

First, we build the audit for tabular mixed regression and for mixtures of discrete Markov chains. Each version returns the subspace-angle statistic, the matched component error, and a per-component report. The report is meant to support statements such as "component 2's weight dropped from 0.3 to 0.05" or "components 1 and 3 merged." The number of components $k$ is chosen by the eigenvalue gap of $M_R$, and we report how sensitive the audit is to that choice. The file flags the choice of $k$ as heuristic in Markovletics, and we do not claim to solve it.

Second, we calibrate the audit. Under the null that the generator is faithful, the statistics should match their real-versus-real bootstrap distribution. We also want sample efficiency, so the subspace test should detect blended mechanisms at sample sizes where a classifier two-sample test is still near chance. Theory predicts that subspace estimation needs on the order of $d$ samples, since the moment matrix concentrates at rate $\sqrt{d/N}$ under the covariate assumptions of the mixed-regression literature. A flexible discriminator over $(x,y)$ has no such targeted rate.

Third, we test whether the audit predicts usefulness for heterogeneity-sensitive downstream tasks. These tasks fit per-subgroup models on synthetic data and evaluate them on real held-out rows with known component labels. We compare the audit's ability to predict that usefulness against existing fidelity scores.

Baselines

The naive reference point combines column-wise Kolmogorov–Smirnov or chi-squared tests with a comparison of pairwise correlation matrices, run on exactly the same real and synthetic samples. On planted mixtures with symmetric components we expect it to rate blended generators as perfect. Beating it means flagging those generators while it does not.

The strongest existing audit is the probabilistic discriminator of Synthetic Tabular Data Validation, alongside a plain classifier two-sample test using gradient-boosted trees. Both are trained on the same real and synthetic rows, with the same held-out split and a hyperparameter search budget at least as large as the time our audit takes. No reported result for this discriminator on Adult or the SDV tables was found on Lacuna; the file states convergence to Monte Carlo ground truth without numbers. We therefore reproduce it ourselves. Beating it means two things. The first is higher detection power at matched false-positive rate for $N \le 2{,}000$ real rows. The second is a higher rank correlation with downstream per-subgroup error across generators.

We also compare against the leave-one-out and joint tiers of Structured Evaluation of Synthetic Tabular Data, run with its released metric definitions on the same data.

For the recovery engine, the reference numbers come from Learning Mixtures of Markov Chains and MDPs. On the 8×8 Gridworld, the spectral pipeline with EM refinement reaches 96.6% average clustering accuracy against 73.2% for randomly initialized EM. We must reproduce roughly this gap before trusting the sequential audit.

Two ablations isolate the key ingredient. One replaces spectral initialization with random-start EM, which tests whether reliable recovery, rather than any mixture fit, drives the audit. The other replaces conditional mixed regression with a Gaussian mixture fitted to the joint $(x,y)$, which tests whether conditional structure is what matters.

Why not just…?

Why not just train a real-versus-synthetic classifier? With enough data and a flexible enough classifier, a two-sample test will eventually detect a blended mechanism, and we concede this. The case for the audit rests on two things. The first is sample efficiency, given that Synthetic Tabular Data Validation itself reports bias at small sample sizes. The second is diagnosis, since a classifier score does not say which subpopulation vanished. The power-versus-$N$ experiment settles the first point. If the classifier matches our power at every $N$ we test, the contribution shrinks to interpretability, and we will report that.

Why not just fit a Gaussian mixture to the joint data on both sides and compare clusters? This is the most serious threat. For Gaussian covariates, a mixture of regressions is a zero-mean Gaussian mixture in $(x,y)$ whose components differ only in covariance, so a full-covariance mixture model can in principle represent it. The model needs on the order of $d^2$ parameters per component, against $d$ for mixed regression, and it fits by EM with random starts. The Gridworld gap between 73.2% and 96.6% suggests such fits are unreliable. We run this as the second ablation and sweep $d$ from 5 to 100. If the joint mixture audit matches ours at moderate $d$, we will narrow our claim to the high-dimensional regime.

Why not just evaluate train-on-synthetic, test-on-real accuracy? A single global model trained on blended synthetic data and one trained on faithful synthetic data can score the same, because neither captures opposite-sign components. Standard downstream scores therefore miss the failure. Is this Model for Real? makes the related point that predictive accuracy does not certify that the mechanism is authentic. We include per-subgroup downstream evaluation as the target the audit should predict, not as a competitor.

Proposed Experiments

Semi-synthetic tables with known ground truth. We take real covariates from the Adult 10,000-row subset and the SDV Student, Expedia and Census tables, and plant outcomes from $k \in \{2,3,5\}$ regression components with varying separation and weights. We train the generators studied in Structured Evaluation of Synthetic Tabular Data, namely CTGAN, TVAE, synthpop and PCC. We add three controls: a Gaussian mixture synthesizer, a deliberately blended Gaussian-copula generator that matches all marginals and pairwise correlations, and an oracle generator.

The decisive check has three parts. On the blended generator, the column-wise, pairwise and joint-tier scores should be high. The classifier two-sample test should sit near chance at $N \le 1{,}000$. The subspace angle should fall outside the real-versus-real null at the same $N$. Across all generators, we test whether $D$ rank-correlates with per-component downstream error more strongly than the discriminator divergence does. If it does not, the central claim fails.

Robustness to junk rows. We inject a fraction of unphysical synthetic rows and compare plain alternating minimization with the trimmed variant from Iterative Least Trimmed Squares. We expect the trimmed audit to separate "the structure was lost" from "the generator emits a corrupted tail."

Sequential arm. On Gridworld we have ground-truth personas. On LastFM listening trails, used in both Learning Mixtures of Markov Chains and MDPs and Markovletics, we have no ground truth and use split-half stability of the recovered components as the reference. We compare a single fitted Markov chain, a recurrent sequence generator, and a mixture-of-chains generator. The audit should rank them by how well they preserve personas, measured as the matched transition-matrix distance.

Results from the real-data arms will be presented as consistency evidence, not as accuracy claims, because no ground-truth component numbers are reported on Lacuna for LastFM.