Seed-Free Convex Probes for Measuring What Synthetic Data Is Worth

Teams that train on simulator output keep asking the same question: how much real data can this synthetic set replace, and which parts of it actually help? Today the only trusted answer comes from end-to-end retraining. You retrain the full tracker or detector for every synthetic subset and every mix ratio, sometimes against several real benchmarks, and then compare scores that also move around with random seeds. We propose a cheaper measuring instrument. Freeze a generic image encoder. On top of it, train a small convex model, meaning one with a single optimum, such as a logistic classifier for detection crops or a pairwise identity-matching head for tracking. Solve it to a certified optimum with deterministic cyclic incremental methods. These are optimizers that visit each block of data in a fixed round-robin order and keep a per-block memory of gradients and curvature. Because the answer is a unique optimum, the measurement has no seed noise. Because that memory decomposes by block, removing or adding one synthetic scenario family and re-solving costs a few passes, not a fresh training run.

The claim is that this probe gives block-level substitutability curves and synthetic-to-real equivalency ratios whose rankings match expensive end-to-end experiments. The test is whether it ranks synthetic scenario families the way full retraining does. One example is the single-person versus multi-group crowds of the M3Act generator, where full retraining showed the former hurts DanceTrack and the latter helps. Another is Virtual KITTI versus Virtual KITTI 2. The probe should do this at a small fraction of the compute, and better than visual-fidelity scores or single-point gradient similarity.

Core Claim

Synthetic-data evaluation is currently expensive and noisy for two separable reasons. It requires full retraining. It also compares single stochastic runs, so small but real quality differences between synthetic subsets drown in seed variance.

On the Equivalency, Substitutability, and Flexibility of Synthetic Data measures substitutability by retraining MOTRv2-style trackers at several real-data fractions. It reports that 60–80% of real data can be replaced, with a frame equivalency ratio near 30:1. Synthetic Dataset Evaluation Based on Generalized Cross Validation builds normalized transfer matrices from YOLOv5s models retrained across Virtual KITTI, KITTI and BDD100K. It explicitly notes that the coupling to specific architectures and the retraining cost are limitations. Both papers motivate this proposal: they show that task-grounded utility is the right target, and that measuring it is the bottleneck.

Our claim is that a converged convex probe recovers the ordering of synthetic subsets and mixtures that these end-to-end protocols produce. It does not need to recover their absolute numbers. Deterministic cyclic incremental methods make the probe cheap to edit, block by block.

The role of cyclic scheduling here is specific, and we want to be precise about it. At convergence the probe's answer does not depend on update order, because the problem is convex. Cyclicity does not change what is measured. It changes how cheaply and exactly many related measurements can be made.

The supporting results come from the optimization literature. An incremental quasi-Newton method with a local superlinear convergence rate shows that a round-robin pass bounds the age of every stored per-component gradient and curvature estimate by one epoch. That bound is what gives IQN its local superlinear rate at per-step cost independent of the number of components. Surpassing Gradient Descent Provably uses the same bounded-age property for the DIAG method's linear rate. A Superlinearly-Convergent Proximal Newton-type Method for the Optimization of Finite Sums extends it to sparse (L1-regularized) objectives. Celer shows that cyclic coordinate sweeps produce regular dual iterates that can be extrapolated into early, certified duality-gap bounds. It also reports that shuffled sweeps break this extrapolation. These papers supply the method. None of them uses it to evaluate data.

Research Program

The probe treats a synthetic corpus as $K$ blocks $B_1,\dots,B_K$. Each block is one scenario family from the generator, for example "single walker", "multi-group with identity crossings", "night lighting" or "high clutter". A real training subset $R_\rho$ holds a fraction $\rho$ of the available real data. The probe objective is

$$F_S(w) = \sum_{k \in S} \lambda_k \, \ell_{B_k}(w) + \ell_{R_\rho}(w) + r(w).$$

Here $w$ are the convex head's weights. $S$ is the set of synthetic blocks included. $\ell_{B}$ is the average convex loss on block $B$ computed on frozen features. $\lambda_k$ weights block $k$ by its size, and $r$ is an L2 or L1 regularizer.

The probe's score for a configuration $(S,\rho)$ is the real held-out task metric of the minimizer $w^*_S$. That metric is detection AP50 for the KITTI testbeds, and pairwise identity-association accuracy for the tracking testbeds. We read off three quantities:

  • Substitutability curve: the score as a function of $\rho$ for fixed $S$.
  • Equivalency ratio: the number of synthetic samples needed to match the score gained from one additional real sample.
  • Block utility: the change in score when block $k$ is removed from $S$.

The computational trick is decomposable memory. Cyclic IQN, or NIM when $r$ is L1, stores for each block $k$ its gradient and a curvature summary, at most one cycle stale. Removing block $k$ means subtracting its stored terms from the running aggregates and running a few more cyclic passes. Each pass exploits the superlinear local rate from the already-near-optimal start. Adding a block, or changing $\rho$, is symmetric. Celer-style dual extrapolation provides a stopping certificate, so every reported score comes with a bound on how suboptimal its solution is.

The result is that a full leave-one-block-out table and a substitutability curve with $M$ mix ratios cost roughly one solve plus $K+M$ short re-solves. A naive approach needs $K+M$ independent trainings, each carrying seed noise. Memory is $O(Kp)$ for $K$ blocks and $p$ head parameters. This sidesteps the $O(np)$ per-sample memory problem flagged for DIAG, because we store per block, not per sample.

The probe heads are chosen to see the factor that matters. For the tracking testbeds, the M3Act result shows that behavioral structure, not photorealism, decides transfer: single walkers hurt, while crowded multi-group scenes with frequent identity crossings raised DanceTrack HOTA from 68.8 to 73.5. A crop-level person classifier would be blind to this. So the main tracking probe is a convex pairwise identity-matching head: a logistic loss on a bilinear similarity between frozen embeddings of crops from nearby frames, with linear motion-offset features appended. Identity crossings then appear as hard negative pairs. For the driving testbeds, the head is a linear per-crop classifier on frozen detector-backbone features.

The research has three stages. The first builds the probe and its block-edit procedure and verifies exactness. Edited solutions must match from-scratch solves to the certified tolerance. The second measures agreement between probe-derived rankings and end-to-end rankings on the testbeds below. The third measures the cost and variance advantage against the alternatives in the next two sections.

Gap 3 in the research file raises a concern: cyclic updates might overfit to deterministic simulator quirks. That concern does not apply here, because the probe is an evaluation instrument and the final models are still trained however practitioners prefer. We nonetheless check order-independence directly by solving the same probe under shuffled schedules.

Proposed Experiments

Testbed 1: M3Act synthetic data. It is split into its scenario families, paired with MOT17 and DanceTrack. Ground truth comes from end-to-end results in On the Equivalency, Substitutability, and Flexibility of Synthetic Data. On MOT17, 40% real data after synthetic pre-training exceeds 100% real, 20% real nearly matches it, and the equivalency ratio is about 30:1. On DanceTrack, multi-group data raises HOTA from 68.8 to 73.5, while single-person data degrades it. We will additionally retrain a smaller tracker end to end, with several seeds, for every leave-one-family-out configuration we can afford. This gives a ranking with error bars to compare against. DanceTrack documents that MOT17-tuned trackers fall from roughly 60–70 to 40–50 HOTA on DanceTrack. This makes DanceTrack the setting where a probe that only measures appearance should fail, and where the identity-matching head must earn its keep.

Testbed 2: driving. It uses Virtual KITTI and Virtual KITTI 2 against KITTI and BDD100K, reproducing the generalized cross-validation protocol. The reference points are $A_o=0.49$ and $S_o=0.45$ for Virtual KITTI from Synthetic Dataset Evaluation Based on Generalized Cross Validation. Virtual KITTI 2 reports roughly a 50% smaller sim-to-real gap than its predecessor, with tracking at 91.2% MOTA on the synthetic clone against 91.5% on real KITTI. A correct probe must rank Virtual KITTI 2 above Virtual KITTI for transfer to real KITTI. It should also give probe-based analogues of $A_o$ and $S_o$ whose ordering across the three datasets matches the end-to-end matrix.

Testbed 3: synthetic tumor CT, secondary. It uses synthetic tumors evaluated on LiTS and FLARE'23. The question is whether probe block utility identifies the tumor-size regimes that Synthetic Data as Validation found most valuable. In that paper, tiny-tumor sensitivity rose from 33.9% to 52.3% out-domain. We do not have block-wise end-to-end ground truth for this testbed, so we treat it as qualitative.

Sanity check. It uses MNIST logistic regression with synthetically corrupted blocks of known harm, such as label noise and feature shift. Ground-truth block utility is cheap to compute exactly there. Edit exactness and ranking can be verified without any perception pipeline.

Metrics. The first is Spearman rank correlation between probe and end-to-end orderings, over blocks and over mix ratios. The second is the probe's estimated equivalency ratio and the real-data fraction at which its substitutability curve crosses the 100%-real reference. The third is total compute, in GPU-hours and in gradient passes, against the end-to-end protocol. The fourth is run-to-run variance, which is zero for the converged probe by construction, compared with the seed variance of the alternatives at equal compute.

Baselines

Naive reference: visual-fidelity scoring. It ranks synthetic blocks by their Fréchet Inception Distance to the real training set. Fréchet Inception Distance is a distance between feature statistics of two image sets. It is the cheapest common practice, and the probe must beat it to justify itself. It is run on the same blocks and real reference images. Synthetic Dataset Evaluation Based on Generalized Cross Validation states that such metrics often fail to track downstream utility. The M3Act photorealistic-single-walker result suggests it will mis-rank DanceTrack blocks. No reported result found on Lacuna for an FID-based ranking on these testbeds. Beating it means higher Spearman correlation with the end-to-end ordering on both MOT17 and DanceTrack.

Strongest existing method: end-to-end substitutability and cross-validation. This is the M3Act protocol for tracking and the generalized cross-validation protocol for driving. It is not a competitor to beat on accuracy. It is the ground truth the probe tries to reproduce at lower cost. Its reported numbers are the targets listed above: 40% real exceeding 100% real on MOT17, an equivalency ratio of about 30:1, HOTA rising from 68.8 to 73.5 on DanceTrack with multi-group data, and $A_o=0.49$, $S_o=0.45$ for Virtual KITTI. "Beating" it means matching its ranking, with a Spearman correlation whose confidence interval overlaps the seed-to-seed agreement of the end-to-end protocol with itself. We estimate that agreement by rerunning the protocol with several seeds. The probe must do this at least an order of magnitude cheaper, measured in total GPU-hours under the same hardware.

Strongest cheap competitor: a stochastic linear probe. This is the same convex head trained with ordinary minibatch SGD and random reshuffling for a fixed epoch budget, repeated over seeds. It is given the same frozen features and the same total gradient-evaluation budget as our probe including all block edits, and its learning rate is tuned on the same held-out split. It isolates whether certified convergence and decomposable memory matter, or whether any cheap linear probe suffices. Beating it means lower variance of block-utility estimates at equal compute, and higher rank agreement with the end-to-end ordering.

Ablation of the scheduling ingredient. We run the identical IQN and NIM solvers under random-permutation and with-replacement block sampling instead of round-robin. The final solutions should coincide, since the problem is convex. The ablation measures what the cyclic order buys in passes-to-certificate and in edit accuracy, because staleness of stored block information is no longer bounded by one cycle. Randomness and permutations in coordinate descent methods gives the expectation: on diagonally dominant quadratics, cyclic sweeps beat random permutations, which are themselves 1.5–1.71 times faster than with-replacement sampling. Celer reports that shuffling degrades dual extrapolation. If the ablation shows no gap on our probes, the cyclic component is unnecessary and we will say so.

Influence-style competitor: gradient alignment. It scores each block by the inner product between its mean gradient and the real-validation gradient at the joint optimum, optionally preconditioned by a conjugate-gradient Hessian solve. It is run on the same converged head. No reported result found on Lacuna for gradient-alignment ranking of synthetic data on these benchmarks. Beating it means better agreement with exact leave-one-block-out effects, especially for large blocks, where first-order approximations should break down.

Why not just…?

Why not just use full-batch L-BFGS, warm-started for each edit? This is the most serious threat. On a convex head, full-batch L-BFGS is also deterministic and seed-free. Warm-starting it from the previous optimum after removing a block may converge in a handful of iterations. If it matches our edit cost, the cyclic-incremental part of the proposal reduces to an implementation detail, and the contribution becomes "a converged convex probe predicts synthetic utility". That would still be useful, but it is narrower. We will run it as an additional baseline with identical tolerances and report passes and wall-clock per edit. Our expected advantage rests on two things: the superlinear incremental rate reported by An incremental quasi-Newton method with a local superlinear convergence rate, and the fact that block removal in IQN subtracts exact stored curvature rather than discarding L-BFGS's history. Whether that advantage is large is an open empirical question, and the experiment settles it.

Why not just retrain end to end on fewer configurations? Practitioners already do this, and it is the ground truth. However, the number of configurations grows with the number of scenario families times the number of mix ratios. Each configuration also needs several seeds before a difference like the DanceTrack single-walker penalty can be trusted. The probe is meant to screen this space and send only the top candidates to full retraining. It fails its purpose if its screening ranking disagrees with end-to-end rankings on the M3Act and KITTI testbeds, which is exactly what we measure.

Why not just score blocks by FID or another fidelity metric? The research file gives direct evidence against this as a utility measure. Synthetic Dataset Evaluation Based on Generalized Cross Validation reports the poor correlation. The M3Act result shows that interaction structure, which FID cannot see, decides DanceTrack transfer. We still run FID because it is the honest zero-cost reference.

Why not just interleave synthetic and real batches cyclically during training and skip the probe? This is the research file's Gap 2. It changes training rather than measurement, and it inherits the open worry that deterministic orderings amplify simulator bias. The bounded-staleness theory that would justify it is only established for convex objectives. We regard it as a follow-up that the probe would inform, by telling us which blocks deserve slots in the cycle. It is not a substitute for knowing which blocks help.

Decisive Checks

The proposal stands or falls on four observations.

The first concerns DanceTrack. The probe must assign negative or near-zero utility to the single-walker family and clearly positive utility to the multi-group family, matching the 68.8-to-73.5 HOTA result. If the identity-matching head cannot see this, frozen-feature probes are blind to behavioral complexity, and the approach is limited to appearance-driven domains.

The second concerns the MOT17 substitutability curve from the probe. Its crossing of the 100%-real reference should fall in the same region as the reported 20–40% real fraction. The two curves need not agree exactly.

The third concerns driving. The probe must rank Virtual KITTI 2 above Virtual KITTI for transfer to KITTI, and its cross-dataset ordering must match the generalized cross-validation matrix.

The fourth concerns cost and exactness. Block edits must reproduce from-scratch solutions to the certified tolerance at a small fraction of their cost. The cyclic-versus-shuffled ablation and the warm-started L-BFGS comparison must show whether round-robin scheduling is doing real work. If the first three hold but the fourth shows no scheduling advantage, we will report the probe as the contribution and drop the claim about cyclic methods.