Synthetic training data is cheap to make and expensive to vet. The most task-faithful way to vet it is the generalized cross-validation idea in Synthetic Dataset Evaluation Based on Generalized Cross Validation. That paper trains a detector on the synthetic set and on each real reference set, then scores every model on every test split. The cost is one full training run per dataset. That is affordable once, but not when a team wants to compare dozens of generator settings, weather mixes, or synthetic-to-real ratios. The cheap alternative, a distribution distance such as the Fréchet Inception Distance (a single number comparing feature statistics of two image sets), is exactly what that paper argues is not task-relevant.
This proposal asks whether a cheap, training-free score computed in a frozen image encoder's feature space can reproduce the ranking that the expensive cross-validation matrix produces. The specific bet is about what a useful score must ignore. Plain feature distances mix two different gaps. One is a "style" gap: rendering, texture, and sensor look. The other is a "content" gap: whether the synthetic objects cover the shapes, poses, and contexts real objects take. A detector partly absorbs the first and cannot recover from the second. We therefore estimate a domain direction in feature space, project it out, and measure per-class coverage of real data by synthetic data in what remains. Landmark kernel approximations keep this tractable at millions of object crops. The deliverable is an empirical answer, with costs, to the question of whether this score is a valid substitute for the cross-validation matrix, and whether it does better than simpler substitutes.
Core Claim
Rankings of synthetic dataset variants by cross-dataset transfer quality (the simulation-quality score $A_o$ and transfer-quality score $S_o$ of the generalized cross-validation paper) can be predicted by a domain-residualized, class-conditional coverage score at a small fraction of the cost. The score should beat both raw feature distances and a no-projection ablation. The claim is falsifiable in a specific way. If a linear probe trained on the same frozen features and cross-evaluated across datasets predicts the rankings equally well at similar cost, then the geometric machinery adds nothing, and we will report that as the result.
What Prior Work Gives and Leaves Open
Two papers motivate the gap. The generalized cross-validation paper supplies the ground-truth notion of quality we try to predict. It builds an $(N+1)\times(N+1)$ matrix of normalized cross-dataset scores from YOLOv5s detectors trained on Virtual KITTI, KITTI and BDD100K. It reports $A_o = 0.49$ and $S_o = 0.45$ under AP50 for Virtual KITTI. It never asks whether the matrix can be predicted without training.
Deflating Dataset Bias Using Synthetic Data Augmentation varies synthetic-to-real ratios at fixed volume. It finds a cross-dataset optimum near 50–60% synthetic, which raised parking-slot cross-dataset F-measure from 0% to 32.4%. It explicitly leaves the reason for that optimum unexplained. A coverage score that peaked at the same ratio would offer a mechanistic account. That makes the result a sharp test for us.
Two papers supply methods. The first is Enhancing Compositional Generalization via Compositional Feature Alignment. It trains a class head and a domain head on frozen features under an orthogonality constraint $W_1 W_2^\top = 0$, then fine-tunes the encoder. We reuse only its first stage, as a way to find the domain subspace. We never fine-tune, because our goal is measurement, not better generalization.
The second is FastMap, MetricMap, and Landmark MDS are all Nystrom Algorithms. It shows that landmark methods are low-rank Nyström approximations of a Gram matrix (the matrix of all pairwise similarities). It also warns that non-Euclidean, indefinite similarity matrices break some of them. That warning constrains our design: we use positive-definite Gaussian kernels on Euclidean features, and we use Landmark-style Nyström rather than FastMap-style deflation.
Two papers constrain the scope. On the Equivalency, Substitutability, and Flexibility of Synthetic Data finds that, on DanceTrack, behaviorally complex synthetic scenes raised tracking HOTA from 68.8 to 73.5, while visually realistic but behaviorally simple ones hurt. A per-frame appearance score cannot see behavior. We therefore treat that setting as an expected failure case, not a success target.
AutoEval Done Right: Using Synthetic Data for Model Evaluation combines many cheap noisy labels with a few trusted ones to obtain unbiased estimates. Its guarantees assume the cheap and trusted samples come from the same distribution. We borrow this logic in a secondary analysis and flag where that assumption is strained.
The Score
Let $f$ be a frozen encoder, such as DINOv2 or CLIP, applied to object crops. We train the two orthogonal heads on crops pooled from all datasets, using class labels and dataset-of-origin labels. Let $P$ be the projector that removes the span of the domain head's rows. For class $c$, let $S_c$ be the synthetic crops and $R_c$ the crops from a real reference set. The score has two parts, combined across classes by real-data class frequency.
The first part is a projected kernel distance $\mathrm{MMD}_k(P f(S_c), P f(R_c))$. MMD (maximum mean discrepancy) is the standard kernel two-sample distance, here with a Gaussian kernel $k$. It is computed through a Nyström approximation with $m$ landmarks, which reduces the cost from quadratic in the number of crops to $O(mN)$.
The second part is a recall term: the fraction of real crops whose projected feature lies within the $k$-th nearest-neighbor radius of some synthetic crop. It measures whether synthetic data reaches the regions where real objects actually sit.
The working hypothesis is that recall after projection drives $S_o$, and that the projected distance to the closest real set drives $A_o$.
Research Program
We will build a small modular engine in the style of Universal Causal Evaluation Engine. That engine separates data generation from estimators behind a fixed fit/predict interface. Ours has four swappable parts.
- A dataset adapter maps each source's labels onto a shared taxonomy and emits object crops.
- An embedder wraps a frozen encoder.
- A proxy computes a score from embeddings.
- A ground-truth runner trains the detector and fills in cells of the cross-validation matrix.
The point of the modularity is narrow and practical. Every proxy, including the baselines, sees identical crops, identical embeddings, and identical label harmonization. Differences in correlation then come from the score itself, not from pipeline accidents. Following DeepDIVA, every run logs commit hash and seeds. Because that paper found cuDNN non-determinism survives seeding, we will report seed variance of the ground truth itself. That variance sets the ceiling on achievable correlation.
The ground truth is a population of synthetic variants built from Virtual KITTI. The variants include single-weather and single-lighting subsets, a sweep of subset sizes, and mixtures with KITTI at synthetic ratios from 0% to 100% at fixed total volume. For every variant we train YOLOv5s exactly as in the generalized cross-validation paper. We compute normalized $A_o$ and $S_o$ against KITTI and BDD100K, using three seeds per variant. This expensive matrix is what every cheap score is judged against.
A second population checks generality outside driving. Here we use CG-Bench's source datasets (OfficeHome, DomainNet) from the compositional alignment paper, treating rendered domains such as clipart and sketch as stand-ins for synthetic data and photo domains as real. Ground truth there is classifier transfer. The M³Act, MOT17 and DanceTrack setting serves as the declared failure probe.
Baselines
The naive reference is to predict quality from synthetic dataset size alone. Size is free, and in the ratio sweep it fully determines the synthetic share. Any proposed score must beat its rank correlation with the ground truth, or it is measuring nothing beyond volume.
The standard cheap metric is raw-feature MMD and Fréchet distance between all synthetic and all real crops. We compute these in the same frozen embedding, with the same Nyström landmarks and the same crops. They are the strongest competitor among distribution distances, because they are what practitioners use today. To beat them, our score must achieve a higher Spearman rank correlation with ground-truth $A_o$ and $S_o$ across variants. The gap must be larger than the bootstrap confidence interval taken over variants. No reported result was found on Lacuna for how well these distances correlate with $A_o$ or $S_o$. Establishing that number is part of the contribution.
The strongest competitor overall is a linear-probe cross-validation matrix. For each dataset we train a linear classifier on frozen crop features, cross-evaluate it across datasets, and normalize the scores exactly as the generalized cross-validation paper does. This copies the expensive method's structure but removes encoder and detector training. It gets the same embeddings and a matched hyperparameter search budget, and we report its wall-clock cost next to ours.
The second strong competitor is reduced-budget detector training: YOLOv5s for about 10% of the schedule, which matches our score's GPU-hours. Beating either competitor means higher rank correlation at equal or lower cost. The reference point for what full training yields is the generalized cross-validation paper's $A_o = 0.49$ and $S_o = 0.45$ (AP50, YOLOv5s, Virtual KITTI). We expect to reproduce those values before trusting the rest of our ground truth.
Two ablations isolate the key ingredient. In the first, we run our score with the projector $P$ removed. In the second, we replace $P$ with removal of a random subspace of the same dimension. If neither ablation loses correlation, the claim that style must be separated from content is wrong.
Why not just…?
Why not just use Fréchet distance or MMD in a good embedding? These distances count rendering-style differences as heavily as missing content. The generalized cross-validation paper's motivation is precisely that such scores do not track task utility. What its evidence does not establish is how badly they fail at ranking variants that share one renderer. If raw MMD matches our score on the Virtual KITTI population, the projection is unnecessary there. The ablation with the projector removed tests exactly this.
Why not just cross-evaluate linear probes on frozen features? This is the real threat. It is cheap, it is directly task-shaped, and it may capture everything our geometric score does. Our reason to expect a gap is that crop-level classification ignores localization and context. Recall-type coverage, which asks whether synthetic data reaches all regions where real objects lie, is not something a probe's accuracy reports directly. That is a hypothesis, not evidence. If the probe matrix ties us at comparable cost, we will recommend the probe and report the coverage score as a diagnostic only.
Why not just train the detector briefly? Short schedules may preserve rankings, since early-training transfer often orders datasets correctly. We match its compute to ours. If it wins, the honest conclusion is that cheap training beats geometry, and the engine still delivers that finding.
Why not just use prediction-powered estimation over a few full trainings? The AutoEval approach would combine many cheap proxy scores with a handful of fully trained cells to estimate population-level quantities without bias. One such quantity is the average transfer quality of a generator configuration. We will run this as a complement, not a replacement, for two reasons. It estimates averages, not rankings of individual variants. And its same-distribution assumption is strained when the trusted cells are chosen non-randomly. We will select those cells uniformly at random to keep the assumption plausible.
Decisive Checks
The primary test is Spearman correlation with ground-truth $A_o$ and $S_o$ across the Virtual KITTI variant population, reported against each baseline at matched cost. Seed variance of the ground truth bounds the attainable maximum.
The second check concerns the synthetic-ratio sweep. Does our score peak near the 50–60% optimum reported by the dataset-bias paper, while raw MMD changes monotonically with the synthetic share? A peak only in our score would explain that optimum: mixtures maximize coverage before the style gap dominates.
The third check is generality: the same comparisons on the CG-Bench-derived population.
The fourth is the DanceTrack probe. We predict our score will fail to separate behaviorally simple from behaviorally complex synthetic scenes. Confirming that failure delimits what any appearance-geometry score can certify.
References
- Synthetic Dataset Evaluation Based on Generalized Cross Validation
- Deflating Dataset Bias Using Synthetic Data Augmentation
- Enhancing Compositional Generalization via Compositional Feature Alignment
- FastMap, MetricMap, and Landmark MDS are all Nystrom Algorithms
- On the Equivalency, Substitutability, and Flexibility of Synthetic Data
- AutoEval Done Right: Using Synthetic Data for Model Evaluation
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models
- DeepDIVA: A Highly-Functional Python Framework for Reproducible Experiments