Selecting Synthetic Data Against a Real Reference

Synthetic data is cheap, but a model trained on it can end up worse than a model trained on less real data. On SVHN with only 1,000 labeled images, adding every generated image cut accuracy from 92.0% to 88.1% (Pooladzandi et al.). The usual fixes filter and subsample the synthetic pool, but they leave two numbers to guesswork: how many synthetic samples to keep and how strict the filter should be. They also leave a third question unanswered, namely how far the synthetic data still is from reality.

We propose to choose synthetic data by asking a weighted subset of synthetic samples to stand in for the small real dataset we already have, rather than for the synthetic pool it was drawn from. A coreset, in the classical sense, is a small weighted subset whose losses approximate those of a larger dataset for every model one might evaluate. We keep that guarantee but swap the target: the synthetic subset must reproduce the losses that a family of probe models incurs on the real data. This one change does three things. It selects against generator artifacts instead of preserving them. It gives a stopping rule for subset size that is set by the statistical noise of the real data rather than by hand. And the leftover mismatch, measured before any final training, is a candidate predictor of the real-world gap that synthetic-trained models will show.

Core Claim

Gradient-matching coresets on synthetic pools, such as the CRAIG step in Pooladzandi et al., faithfully summarize the synthetic distribution, including its hallucinations and biases. The entropy filter placed in front of CRAIG is what removes bad samples, and its threshold was tuned manually; the authors list automating it as future work. We claim that redirecting the coreset objective toward a real reference set makes the separate filter, the hand-set threshold, and the hand-set fraction unnecessary.

Let $R$ be the real labeled set, $G$ the synthetic candidate pool, and $\mathcal{H}$ a finite family of probe classifiers. We seek non-negative weights $w$ on a subset $S \subset G$ that minimize

$$D(S,w) = \max_{h \in \mathcal{H}} \frac{\left| \frac{1}{|R|}\sum_{x \in R} \ell(x,h) - \sum_{s \in S} w(s)\,\ell(s,h) \right|}{\frac{1}{|R|}\sum_{x \in R} \ell(x,h)},$$

where $\ell(x,h)$ is the loss of probe $h$ on labeled example $x$. This is the strong-coreset inequality from the coreset literature, restricted to a finite query set and with the real set as the target. The probes are linear heads on frozen features, checkpoints from a cheap training run, and class-conditional variants of both. We fit within each class separately, so that matching cannot be achieved by trading errors across classes.

Because $R$ is small, $D$ cannot meaningfully go below the noise in $R$ itself. We estimate that noise floor $\tau$ by bootstrapping $R$: we compute $D$ between resamples of $R$ and $R$, and take a high quantile. Selection grows $S$ greedily, by a Frank-Wolfe-style step over the probe-loss matrix, until the discrepancy measured on a held-out set of probes falls below $\tau$ or stops improving. The final size $|S|$ is an output of this process rather than an input. If the procedure stalls above $\tau$, the remaining gap $D^\star - \tau$ is our proposed pre-training estimate of the real-world validation gap.

Relation to Prior Work

Two papers motivate the problem. Pooladzandi et al. show both that uncurated synthetic data poisons training and that entropy filtering plus CRAIG recovers to 94.5%. Their coreset target is the synthetic pool, and their knobs are manual; we change the target and remove the knobs. Chang et al. measure substitutability and an exchange rate of roughly 30 synthetic frames per real frame on MOT17, but only by repeated end-to-end training. Our discrepancy curve offers a training-free estimate of that rate, which we can check against their figure.

Two papers constrain the design. Refined Coreset Selection (LBCS) finds the minimal coreset size meeting a performance tolerance. Its inner-loop retraining is expensive, and it was never applied to synthetic pools. We borrow its lexicographic attitude: quality first, then size, with no scalarized size penalty to tune. We replace its retraining loop with a statistical stopping rule. AMH is a reminder that adaptive rules carry their own meta-parameters. Its leak rate worked on BloodMNIST and collapsed CIFAR-10 at 5% data to chance. Our single meta-parameter is the bootstrap quantile, and we will report sensitivity to it explicitly.

Two further sources supply method and evaluation hygiene. ACM adapts inner effort by progress per unit cost; we use the same logic to decide when an extra greedy step is no longer worth its compute. Rosenberg-Kima & Pardos warn that downstream accuracy alone can hide divergence between synthetic and real data. That warning is why we validate the discrepancy against held-out probes rather than only against final accuracy.

Research Program

The system has three swappable parts: a generator, a selector, and a probe family. Each is wrapped behind a small fit/score interface in the spirit of the Universal Causal Evaluation Engine's estimator API, so that generators and selectors can be crossed without rewriting the harness. This modularity serves the evaluation, not the method. It lets us run every selector against every generator under identical seeds and budgets. We log seeds and code versions as in DeepDIVA, and report several seeds because that work found GPU determinism flags insufficient for bitwise reproducibility.

The analysis has three threads. The first tests whether held-out-probe discrepancy actually transfers to unseen models. A selector can match a finite probe set with very few points, by Carathéodory-type arguments, and still fail on new models, so this transfer is the central technical risk. The second tests whether the stopping rule picks sizes near the empirical optimum found by retraining. The third tests whether the stalled residual predicts the real-test accuracy drop across generators of different quality. To span quality, we degrade the generator deliberately: fewer sampling steps, an earlier checkpoint, or added label noise.

Baselines

The do-nothing references are real-only training and all-synthetic augmentation. On SVHN with 1,000 labels, Pooladzandi et al. report these at 92.0% and 88.1% accuracy. We rerun both under our training recipe so that comparisons are internal.

The strongest existing method is the full entropy-filter + CRAIG + pseudo-label pipeline of Pooladzandi et al., with a reported 94.5% SVHN accuracy. It is the strongest competitor because it targets exactly this poisoning problem. We run it on the same generator outputs, labeled set, and student architecture, with pseudo-labeling either on for both methods or off for both. Its entropy threshold and fraction are tuned on the same real validation split and given the same total GPU budget our method uses. Beating it means matching or exceeding its accuracy with no manual threshold. Alternatively, it means matching its accuracy at a selection cost below its tuning cost, across all three seeds-averaged settings.

We also include a validation-tuned grid search, which crosses entropy threshold with subset fraction and retrains for each configuration on the real validation split. This is the honest "just tune it" competitor, and we run it at a matched compute budget.

LBCS, using the released code, is applied to the real-plus-synthetic pool. We expect it to be the most accurate size oracle but also the most expensive. We report accuracy, chosen size, and wall-clock time, and treat its chosen size as a reference point for our stopping rule.

Two ablations isolate the key ingredient. In the first, the same selector targets the synthetic pool's losses instead of the real set's, which essentially reproduces CRAIG's objective. In the second, we keep our target but fix the subset size at the value LBCS chose, which isolates the stopping rule. Uniform random synthetic subsets of the same size complete the picture.

Why not just…?

Why not just tune the entropy threshold and coreset fraction on a real validation set? This is the most serious alternative, and it may well match our accuracy. Its cost is one retraining per configuration, and it yields no estimate of the remaining synthetic-real gap. The matched-compute grid-search baseline settles the accuracy question directly. If grid search wins at equal compute, our contribution narrows to the gap-prediction claim, and we will report it that way.

Why not just run LBCS on the combined pool? LBCS answers the minimal-size question directly but pays for it with inner-loop retraining. Its authors list that compute as an open limitation, and it was not tested on synthetic data. We expect it to be a strong accuracy reference that is too costly at ImageNet-scale pools. The wall-clock comparison will show whether our statistical stopping rule tracks its chosen sizes closely enough to substitute for it.

Why not just pretrain on all synthetic data and fine-tune on real, as Chang et al. do? That recipe works for tracking, but it still requires a meaningful fraction of real data. They needed 20% to match the full-real baseline. It also gives no per-sample control over poisoning. We include pretrain-then-fine-tune as an additional arm on SVHN and MOT17, and will check whether our selection improves the pretraining stage itself.

Proposed Experiments

The primary testbed is SVHN with 1,000 labeled and 64,932 unlabeled images, following Pooladzandi et al.. We use their latent energy-model generator where reproducible, and a standard conditional diffusion generator otherwise, crossed with all selectors. The headline metric is test accuracy against the 94.5% reference; we also report selected size and selection compute.

The stress test is CIFAR-10 at a 5% real fraction, where AMH reports EfficientNet-B0 collapsing to about 10% accuracy. Here we ask whether real-referenced synthetic selection keeps training stable where the optimizer alone did not.

The exchange-rate check uses MOT17 and DanceTrack. If the M3Act synthetic data of Chang et al. is obtainable, we compare the synthetic-per-real ratio implied by our discrepancy curve against their reported ~30:1. We also compare HOTA against their DanceTrack improvement from 68.8 to 73.5.

Two decisive checks determine whether the idea holds. First, across at least five generators of graded quality, the stalled residual $D^\star - \tau$ must rank the downstream real-test accuracy drops correctly; we will report Spearman correlation with seed-level confidence intervals. Second, held-out-probe discrepancy must track discrepancy on freshly trained models. If either fails, the method reduces to another selection heuristic, and we would say so.