Geometry-Diverse Generator Ensembles for Honest Synthetic Data

When people train a model on synthetic data, they usually also judge it on synthetic data, and the two errors compound. Every generator makes systematic mistakes. A model trained on its samples learns those mistakes, and a synthetic test set built by the same generator rewards the model for learning them. The best current remedy, the Deep Generative Ensemble of Synthetic Data, Real Errors: How (Not) to Publish and Use Synthetic Data, trains several generators with different random seeds, trains one downstream model per synthetic dataset, and cross-evaluates each model on the other members' data. Its authors concede the limit: seeds only remove random variation. If every generator in the ensemble shares the same blind spot, the ensemble shares it too.

We propose to vary the geometry each generator is trained under, not only its seed. Banach Wasserstein GAN showed how to train a GAN whose critic (the network that scores real versus fake samples) measures distance in a chosen norm. Depending on the norm, the critic attends to outliers, to coarse global structure, or to fine edges. Each choice should therefore leave a different region of the data poorly modeled. Our bet is that an ensemble whose members fail in different places will cancel shared errors that a seed ensemble cannot reach. That would make synthetic-only performance estimates both more accurate and more honest. Banach Wasserstein GANs have so far been judged only by Inception Score and visual inspection. We would also run the first downstream test of whether their samples are useful to train on.

Core Claim

The benefit of any generator ensemble is capped by how correlated its members' errors are. Consider $K$ downstream models trained on $K$ synthetic sets, whose real-data errors each have variance $\sigma^2$ and pairwise correlation $\rho$. The error of their averaged prediction has variance roughly

$$\rho\,\sigma^2 + \frac{(1-\rho)\,\sigma^2}{K}.$$

This is the standard decomposition for averaging correlated estimators. Adding seeds only shrinks the second term. The shared term $\rho\sigma^2$ stays, and it is exactly the "systemic generator bias" that van Breugel et al. say their method cannot fix.

We claim that the norm used by the critic is a cheap and controllable lever on $\rho$. It works within a single architecture and a single training recipe. Choosing members that span different norms (Lebesgue $L^p$ norms for several $p$, and Sobolev norms $W^{s,2}$, which weight spatial frequencies by a smoothness index $s$) should lower the correlation between members' real-data errors. That in turn should do two things. It should raise real test accuracy, especially on minority classes and subgroups. It should also shrink the gap between what synthetic cross-evaluation predicts and what happens on real data.

The new contributions are three. First, geometry is used as an axis of ensemble diversity. Second, $\rho$ is measured directly as the explanatory quantity. Third, both claims are tested against the obvious competing source of diversity, which is mixing different model families.

Research Program

We would build a single training codebase on the Banach Wasserstein GAN gradient penalty. That penalty constrains the dual norm of the critic's gradient, so swapping the norm means swapping one function. Architecture, optimizer, and step count stay fixed. Following Adler et al., we would use their heuristic rules to set the penalty weight and scale from data norm statistics, so that no member gets extra hand-tuning.

We restrict to norms with $1 < p < \infty$. Those spaces are reflexive and uniformly convex, so the dual map is single-valued and the penalty is well defined. The flat-faced $L^1$ and $L^\infty$ cases make the duality map multi-valued, as discussed in Faces and Support Functions for the Values of Maximal Monotone Operators. We set those cases aside rather than pretend the penalty is well posed there.

Every synthetic dataset is used in two ways. For the training use, a downstream classifier is trained on it and tested on real held-out data. For the evaluation use, the DGE leave-one-out estimate predicts real performance from synthetic data alone. For every ensemble we record three things: real accuracy, the gap between estimated and real performance, and the empirical $\rho$ between members' errors on the real test set. We also record substitutability in the sense of On the Equivalency, Substitutability, and Flexibility of Synthetic Data, meaning the fraction of real training data that synthetic data can replace without loss.

Finally, we would test a member-selection rule that uses no real data. Starting from a pool of geometries, we greedily add the generator whose downstream model disagrees most with the current ensemble on pooled synthetic data. We then check whether this synthetic disagreement tracks the real $\rho$. If it does, practitioners get a usable recipe. If it does not, that is itself a finding about how far synthetic-only evaluation can go.

Baselines

The naive reference point is a single Euclidean WGAN-GP generator, as in Improved Training of Wasserstein GANs. Its downstream model is trained on one synthetic set and scored on a synthetic split from the same generator. This is the practice van Breugel et al. criticize. On their Mexican Covid-19 tabular data it predicted an AUC of 0.841 while the real AUC was 0.738. Any ensemble must beat it on real accuracy and on the size of that estimate gap.

The strongest existing method is the seed-only Deep Generative Ensemble. On the Covid-19 data, van Breugel et al. report a real AUC of 0.942 for DGE with $K=20$, against 0.869 for the naive approach and 0.928 for an oracle trained on real data. Its estimate of real performance was 0.724. We would run it with the same base generator, the same $K$, the same training steps per member, and the same downstream classifier and tuning budget as the geometry ensemble. The two then differ only in what varies across members.

For image data, no reported result was found on Lacuna for downstream accuracy of classifiers trained on Banach Wasserstein GAN or WGAN-GP samples and tested on real CIFAR-10. The only reported figures are generative ones: Inception Score 8.31 for the $L^{10}$ Banach Wasserstein GAN against 7.86 for WGAN-GP. We will establish the downstream numbers ourselves for both.

Beating DGE means two things at equal $K$ and equal compute. Real accuracy must be higher, with worst-class accuracy reported separately. The absolute gap between estimated and real performance must be smaller. Both differences must hold beyond the spread across repeated ensemble draws.

The key ablation separates diversity from simply finding a better norm. For each geometry in the pool we also build a $K$-seed ensemble at that single norm, and take the best of these as an "oracle single-geometry" baseline. If the best single-norm ensemble matches the mixed-geometry ensemble, the gain came from norm choice, not from diversity, and the core claim fails.

Why not just…?

Why not just use more seeds? Under the decomposition above, more seeds cannot remove the shared error term. We test this directly by plotting real error and estimate gap against $K$ for seed ensembles. We expect both curves to flatten, and the flat level should be predicted by the measured $\rho$. If seed ensembles keep improving with no plateau through $K=20$, the shared-error story is wrong for these datasets, and the proposal loses its motivation.

Why not just mix model families, such as a GAN, a VAE, and a score-based diffusion model? This is the real threat. Different families plausibly make different mistakes. Score-Based Generative Modeling through Stochastic Differential Equations alone reaches an FID of 2.20 on CIFAR-10, far beyond any GAN here. We therefore include an architecture-diverse ensemble at matched total GPU-hours, which means fewer diffusion members. We compare its $\rho$, accuracy, and estimate gap with the geometry ensemble, and also test a hybrid that adds geometry variation to the GAN members. If architecture diversity wins outright and the hybrid adds nothing, we will report that geometry is a weaker diversity lever than model family. In that case its remaining value is only that it is cheaper and stays inside one training recipe.

Why not just train one better generator? A diffusion model trained alone may give lower downstream error than any ensemble of weaker GANs. However, van Breugel et al.'s analysis says a single generator still yields an over-optimistic self-evaluation. So we report the single diffusion model's estimate gap alongside its accuracy. The comparison then shows whether quality alone fixes the honesty problem, or whether ensembling is still needed.

Proposed Experiments

The primary image testbed is CIFAR-10. Generators are trained on the training split, ResNet classifiers are trained on synthetic sets, and the classifiers are evaluated on the real test split. Generation must therefore be class-conditional, or labeled by a fixed classifier trained on real data. We report results both ways, because labeling leaks real information.

The geometry pool is $L^p$ for $p \in \{1.5, 2, 4, 10\}$ and $W^{s,2}$ for $s \in \{-1, -0.5, 0.5, 1\}$. It covers the range in which Adler et al. found Sobolev indices between $-1$ and $0$ most stable on CelebA. CelebA with binary attribute prediction is the second testbed. It adds subgroup structure, which lets us test the worry in Gap 1 that norms emphasizing outliers might worsen minority distortion rather than repair it.

The tabular check uses the Mexican Covid-19 data, so that our seed-DGE reproduction can be compared with van Breugel et al.'s 0.942. Only $L^p$ norms on standardized feature vectors are used there, because Sobolev smoothness has no meaning for tabular columns. We would also estimate substitutability on CIFAR-10. This is the smallest real fraction which, combined with synthetic pre-training, matches full real training, following Chang et al.'s protocol.

The decisive check is whether measured $\rho$ explains the outcomes across all ensembles, including seed, geometry, architecture, and hybrid ensembles. A strong, consistent relation between lower $\rho$ and both higher real accuracy and smaller estimate gaps, with geometry ensembles reaching lower $\rho$ than seed ensembles at equal cost, supports the claim. Geometry ensembles that do not lower $\rho$, or a single best-norm ensemble that matches them, would refute it. Either way, the first downstream audit of Banach-constrained generators is a useful result.