Where Synthetic Data Fails: Nonparametric Diagnosis of the Fidelity Paradox

Synthetic tabular data is routinely judged by how closely its overall distribution matches the real one. The judgment is often wrong. A Bayesian network can reproduce feature distributions almost perfectly and still yield a poor classifier when that classifier is trained on its samples and tested on real data. We propose that this "Fidelity Paradox" is not mysterious. Global fidelity scores measure the wrong thing, weighted in the wrong places. A classifier trained on synthetic data inherits the synthetic data's answer to one question: given these features, how likely is each label? The resulting classifier fails on real data only in two cases. Either the synthetic answer differs from the real one near the decision boundary, or the synthetic data leaves regions empty where real data lives.

We build a small, nonparametric score for exactly these two failure modes. It is computed with nearest-neighbour density estimates, meaning estimates of how crowded the data is around a point, read from the distance to its k-th closest neighbour. We also derive from the same estimates a per-region "exchange rate": how many real samples a given synthetic dataset is worth. The note tests whether this score predicts downstream utility better than the metrics in current benchmark suites, and whether it explains why those suites disagree with utility.

Core Claim

Write $\eta_r(x)$ and $\eta_s(x)$ for the probability of the positive label at feature vector $x$ under the real and the synthetic data, and $p_r(x)$ for the real feature density. Classical plug-in classification theory gives the excess error of a classifier that thresholds $\eta_s$, evaluated on real data:

$$ \mathcal{E}(s) = \mathbb{E}_{x \sim p_r}\Big[\,|2\eta_r(x)-1|\;\mathbf{1}\{\mathrm{sign}(\eta_s(x)-\tfrac12)\neq \mathrm{sign}(\eta_r(x)-\tfrac12)\}\Big]. $$

Several things follow from this expression. The expectation is taken over real feature density, not synthetic. Only label-conditional disagreement enters. Disagreement matters only where it flips the decision. A synthetic set can therefore match the joint distribution well under maximum mean discrepancy (MMD, a kernel distance between two sample sets) and still score badly on $\mathcal{E}$. Conversely, it can mismatch the marginals in regions the classifier never needs and remain useful.

Our claim is that a finite-sample estimate of $\mathcal{E}$ predicts train-on-synthetic, test-on-real utility substantially better than global fidelity metrics. It has two parts, a covered-region disagreement term and an uncovered-mass term, and both are estimated with adaptive-bandwidth nearest-neighbour estimators. It should also predict at least as well as the "fraction of easy samples" heuristic, and it says where the synthetic data fails.

Relation to Prior Work

Two papers motivate the gap. Reimagining Synthetic Tabular Data Generation through Data-Centric AI documents the Fidelity Paradox on 11 OpenML datasets. It then abandons distributional explanations in favour of sample-profiling tools (Cleanlab, Data-IQ, Data Maps) and shows that the proportion of "easy" synthetic samples predicts AUC better than KL or MMD. We read "easy" as "far from the boundary, where $\eta_s$ is sharp." Our hypothesis is that the easy-fraction is a coarse proxy for the boundary-weighted quantity above, and that estimating that quantity directly will predict better and explain why. Synthcity offers more than 25 metrics across fidelity, utility, privacy and fairness. It gives no principle for resolving their disagreements. It reports that adversarial random forests achieve the best quality (0.5475) and detection (0.6721) scores. Our score is meant to be that principle for the fidelity–utility pair.

Two papers supply the method. A Kernel-Based Calculation of Information on a Metric Space shows that a kernel density estimate whose bandwidth is set by probability mass converges to a k-nearest-neighbour estimator. It also reports much lower error than histograms in 10 dimensions with 10 trials per stimulus (0.139 versus 0.876 bits). This justifies our choice of mass-adaptive neighbourhoods over fixed-bandwidth kernels, given that we have moderate dimension and few samples per region. A comparison of likelihood-free methods with and without summary statistics compares simulated and observed data using full-data kernel likelihoods and distances. It flags their quadratic cost. We use approximate neighbour search to stay near $O(n \log n)$, and we apply the full-data idea to generative-model evaluation rather than parameter inference.

One paper constrains the design. Kernel mean matching enhances risk estimation under spatial distribution shifts shows that pointwise density ratios from kernel density estimation explode under support mismatch, with percentage errors above 500%. We therefore never form the ratio $p_r/p_s$. Coverage is measured by comparing a real point's distance to its k-th synthetic neighbour against its distance to its k-th real neighbour. This ratio of radii is bounded and can be clipped.

The closest bridge between the two directions is Differentially Private Synthetic Data with Private Density Estimation. It uses a private kernel density estimate as a reference measure to generate synthetic data, and explicitly leaves downstream utility unevaluated. We do the reverse: density estimation as an evaluator of utility. Its private generator is a natural test subject for us.

Research Program

The first component is the estimator. For each real test point $x_i$, we take its k nearest neighbours among synthetic points, with k chosen by mass. This gives $\hat\eta_s(x_i)$. We get $\hat\eta_r(x_i)$ from neighbours in a held-out real reference split, which avoids using the point itself. The decision-flip term is a soft version of $\mathcal{E}$, with the indicator replaced by a smooth function so that estimation noise near $\eta=1/2$ does not dominate. The uncovered-mass term is the fraction of real points whose synthetic-neighbour radius exceeds their real-neighbour radius by a factor $\tau$. For those points the synthetic classifier is extrapolating. Features are standardised, and categorical columns use a mixed distance. As a secondary condition we also run in the embedding space of a tabular encoder, to check sensitivity to the choice of metric, a known weakness of nonparametric estimators in metric spaces.

The second component is the exchange rate. We fit a learning curve for a real-data nearest-neighbour classifier: excess error as a function of the number $m$ of real training samples. A synthetic set's effective real sample size $m^*$ is the $m$ at which that curve equals the synthetic set's estimated $\mathcal{E}$. Computed per region of feature space, this gives a map. It answers the question On the Equivalency, Substitutability, and Flexibility of Synthetic Data left empirical: that paper finds roughly 30:1 synthetic-to-real for tracking frames, with no theory behind the number. We do not claim a closed-form exchange rate. We claim a measurable one, from the same estimates, which can be checked against real substitution experiments.

The third component is a controlled mechanism test. We construct synthetic sets by perturbing real training data in two matched ways that leave global MMD and marginal KL nearly equal. In one, we resample labels near the real boundary. In the other, we distort feature marginals far from it. If our account of the Fidelity Paradox is right, utility should collapse only under the first perturbation. Our score should track that collapse, and the global metrics, by construction, should not.

Baselines

The naive reference point is reporting no fidelity metric at all and predicting every generator's utility by its dataset-average rank. Any proposed score must beat this, or it carries no information.

The standard fidelity baselines are the global metrics in Synthcity: MMD, Wasserstein distance, Jensen–Shannon divergence, and the detection score from a classifier trained to tell real from synthetic. All are computed through the Synthcity implementation on identical real/synthetic splits. These are the metrics the paradox was observed against.

The strongest existing competitor is the easy-sample proportion from Reimagining Synthetic Tabular Data Generation through Data-Centric AI. It is the only metric in the file shown to predict downstream AUC better than statistical fidelity. We rerun its profiling pipeline with its default settings on the same splits. We found no reported correlation value for the easy-fraction predictor on Lacuna. The paper reports utility gains from data-centric processing of up to 1.64% AUC and model-ranking gains of 4.66% to 92%, but not a correlation coefficient. We therefore recompute it ourselves.

The two most direct competitors are cheap proxy utility and a label-aware discriminator. The cheap proxy is train-on-synthetic, test-on-real with logistic regression, used to predict the utility of other model families. The label-aware discriminator is a two-sample classifier trained on concatenated (features, label) pairs.

Two ablations isolate our ingredients. The first replaces real-density weighting with synthetic-density weighting. The second drops the decision-flip weighting and uses unweighted posterior disagreement. A third ablation estimates the same quantity with gradient-boosted classifiers instead of neighbours, to separate the value of the target quantity from the value of the nonparametric estimator.

Every method gets the same real reference split and the same synthetic sample. Each gets one hyperparameter search of matched size: k and $\tau$ for us, discriminator capacity for the classifier baselines. That search is tuned on a disjoint set of datasets.

Beating a baseline means a higher Kendall rank correlation with held-out utility, averaged across datasets, with a paired bootstrap interval that excludes zero. Held-out utility is real-test AUC of XGBoost and an MLP, models never used to build any score.

Why not just…?

Why not just run train-on-synthetic, test-on-real with a cheap model and skip the density estimation? This is the most serious threat. Logistic-regression utility may already rank generators well for XGBoost. It cannot say where the synthetic data fails or what it is worth in real samples. It is also a single model's view of $\eta_s$, and will be biased toward linear structure. We test this directly. If cheap proxy utility matches our score's rank correlation, the scalar-prediction claim fails, and the contribution narrows to the localisation map and exchange rate. Those are then judged on their own by the substitution experiment below.

Why not just use the easy-sample fraction, which already beats fidelity metrics? Its profiles come from training dynamics, not the real data distribution. Nothing in it weights errors by where real data lives, so a generator that is confidently wrong about labels can look "easy." The boundary-perturbation experiment separates the two directly: boundary label flips that stay confident should fool the easy-fraction but not our score.

Why not just train a discriminator on (features, label) pairs? A discriminator detects any difference, including harmless marginal distortions far from the boundary. This is exactly the failure mode of global metrics. The second matched perturbation should inflate its score without hurting utility. The classifier-estimator ablation keeps its estimation power while using our target quantity. If that ablation matches neighbours, we will report that the quantity, not the estimator, is the contribution.

Proposed Experiments

The main grid follows both benchmark papers: the 11-dataset OpenML suite from the data-centric benchmark and the 18 OpenML datasets plus SIVEP-Gripe from Synthcity. Generators are Bayesian networks, CTGAN, TVAE, normalising flows, TabDDPM and adversarial random forests, plus the private density-estimate generator at several privacy levels, each with five seeds. The primary metric is per-dataset Kendall correlation between each score and held-out XGBoost and MLP AUC. A secondary metric is top-1 generator selection accuracy. On SIVEP-Gripe we also compute scores within minority cohorts. Synthcity reports augmentation raising minority-group AUROC from 0.3244 to 0.7241 for neural nets, so we ask whether our coverage term anticipates which generators help there.

The decisive checks are the following. The matched perturbation study must show a utility collapse that our score tracks while MMD stays flat; if it does not, the account of the paradox is wrong. Ranking must hold up against the easy-fraction and cheap proxy utility on held-out model families. Finally, predicted exchange rates must match real substitution curves. For each dataset we train on synthetic data plus a fraction of real data, following the substitution protocol of the tracking study, and check whether $m^*$ predicts the real fraction at which performance matches the all-real baseline, within a factor of two on most datasets.