Measuring What Synthetic Data Is Worth by Its Gradients

A model trained on synthetic data is a model whose gradients point the wrong way by some amount. Every step of training follows the gradient of the synthetic loss. What matters is how much of that step also lowers the loss on real data. This note proposes to measure that quantity directly, along the path the optimizer actually takes and in the geometry the optimizer actually uses. The motivation is that such a measurement needs only a few hundred real examples, no matter how many parameters the model has. The alternative view, classical uniform convergence (asking the sample gradient to match the true gradient at every possible parameter setting at once), needs sample sizes that grow with network width and exponentially with depth.

The practical payoff is predictive. Today, the "exchange rate" between synthetic and real samples is found by brute force. An example is the finding that roughly 30 synthetic tracking frames are worth one real frame, or that synthetic pretraining plus 20% of the real data matches training on all of it (On the Equivalency, Substitutability, and Flexibility of Synthetic Data). Numbers like these are obtained by retraining on many mixtures, and that paper notes they lack any predictive formula. We aim to predict from a single training run when synthetic steps stop helping, how the answer depends on the optimizer (Adam, Muon, plain SGD), and when to switch to real data.

Core Claim

Consider the error between the gradient a learner computes on synthetic data and the true real-data gradient at the same parameters. This error splits into two parts. The first is a sampling term, which shrinks as more synthetic samples are drawn. The second is a bias term, the gap between the synthetic population gradient and the real population gradient, which no amount of synthetic data removes. Take a normalized steepest-descent optimizer: one that moves a fixed step length in the direction of steepest descent under some norm $\|\cdot\|$, such as sign descent for the $\ell_\infty$ norm or Muon's orthogonalized update for the spectral norm. Write $u_t$ for its update direction, $g_t$ for the synthetic minibatch gradient, $R$ for the real population risk, and $\|\cdot\|_*$ for the dual norm. Its guaranteed first-order progress on real risk at step $t$ satisfies

$$\langle \nabla R(\theta_t), u_t\rangle \;\ge\; \|\nabla R(\theta_t)\|_* - 2\|g_t - \nabla R_{\mathrm{syn}}(\theta_t)\|_* - 2\|\nabla R_{\mathrm{syn}}(\theta_t) - \nabla R(\theta_t)\|_* .$$

This follows from the defining property of the normalized update, $\langle g_t, u_t\rangle = \|g_t\|_*$, together with two triangle inequalities. Two consequences follow. First, real progress stalls once the real gradient norm falls to the level of the bias, so synthetic-only training reaches a floor. Second, the bias is measured in the dual norm of the optimizer, so the same synthetic data can have different value for different optimizers.

Our claim is that the left-hand side can be estimated along the training trajectory with dimension-free sample complexity, meaning the number of real examples needed does not grow with the number of parameters. The estimate would then predict the substitutability curve and the right switch point from synthetic to real data better than distribution-similarity scores or synthetic validation loss. What is new is threefold. The target is trajectory-restricted rather than uniform over all parameters. It is a scalar alignment rather than a vector norm. It explicitly depends on the optimizer's geometry.

Research Program

The theory begins from landscape inheritance. The landscape of empirical risk for nonconvex losses shows that uniform convergence of gradients and Hessians makes empirical stationary points correspond one-to-one with population ones. Empirical Risk Landscape Analysis for Understanding Deep Neural Networks shows the price in deep networks: a rate of order $r^{2l}\sqrt{l\sqrt{\max_i d_i}\,s\log(d/l)/n}$, which is vacuous for wide, deep, dense models. These papers motivate the question and also constrain it. Any guarantee requiring a supremum over all parameters will inherit this cost, so we give up the supremum.

We replace it with a synthetic analogue of landscape inheritance that holds only where the optimizer goes. The first result will be simple. If the bias along the trajectory is at most $b$, then every point where synthetic training stalls has real gradient norm at most about $b$ plus sampling error. If in addition real risk is locally strongly convex with constant $\lambda$ near a minimum, the synthetic minimum lies within roughly $b/\lambda$ of a real one. Here we reuse the local-curvature machinery of Characterization of Excess Risk for Locally Strongly Convex Population Risk. That paper assumes i.i.d. real data and isolated minima, and we inherit its sensitivity to $\lambda$. We will also state honestly that flat, degenerate valleys, the case studied in The Landscape of Non-convex Empirical Risk with Degenerate Population Risk, break the distance statement but not the gradient-level one.

The dimension-free part comes from what we estimate. Estimating the dual norm of the bias directly would reintroduce dimension dependence; the nuclear norm of a noisy matrix difference is biased upward in small samples. Instead we estimate the scalar $A_t = \langle \nabla R(\theta_t), u_t\rangle$, the real descent actually delivered by the synthetic update. We do this by averaging per-example real gradients from a probe set of $m$ real examples, dotted with $u_t$. Each term is a bounded scalar, so Hoeffding's inequality with a union bound over $T$ checkpoints gives error of order $\sqrt{\log T/m}$ with no dependence on parameter count.

To know how much real descent was available, we use a split-probe estimator. One half of the probe set produces the update the optimizer would take on real data. The other half evaluates its alignment. This gives an unbiased estimate of a lower bound on available real progress, which is again a scalar. The ratio $\rho_t$ of delivered to available real progress is our main diagnostic. Summing $A_t$ against the standard descent inequality $R(\theta_{t+1}) \le R(\theta_t) - \eta A_t + \tfrac{L\eta^2}{2}\|u_t\|^2$ turns the diagnostic into a forecast of cumulative real-risk reduction.

The noise model that keeps the sampling term dimension-free comes from A Tale of Two Geometries: Adaptive Optimizers and Non-Euclidean Descent. That paper proves dimension-free rates for Adam, Shampoo, and normalized steepest descent under "adaptive gradient variance," but assumes unbiased gradient oracles. Our setting is precisely the biased-oracle case it leaves out. On the Convergence of Muon and Beyond supplies the spectral-norm geometry for Muon. On the Convergence Rate of LoRA Gradient Descent suggests a low-rank adapter variant, where synthetic fine-tuning is common.

From the decomposition we derive two testable predictions. The first concerns when more synthetic data stops helping. Adding synthetic samples helps until their sampling error falls below the bias. The marginal exchange rate between one real and $k$ synthetic samples is then the $k$ at which the two error sources balance, and it becomes infinite beyond that point. We will fit the sampling and bias scales from early-trajectory measurements and predict the whole substitutability curve. The second is an optimizer-dependence prediction. Synthetic data whose errors are spread across many coordinates but small in each will hurt sign-based methods less than Euclidean SGD. Errors concentrated in a few directions of a weight matrix will hurt Muon less. Finally, we will build a controller in the spirit of the geometric sample-growth scheme of Efficient Nonconvex Empirical Risk Minimization via Adaptive Sample Size Methods. Instead of growing the sample size, it lowers the synthetic fraction of each batch whenever $\rho_t$ falls below a threshold.

Baselines

The naive reference point trains on synthetic data only and on the available real data only, both with the same architecture, optimizer, and tuning budget. This bounds the value any mixing rule can add.

The strongest existing recipe is the pretrain-on-synthetic, fine-tune-on-real protocol of On the Equivalency, Substitutability, and Flexibility of Synthetic Data. On MOT17, it reports that synthetic pretraining plus 20% real data matches a 100%-real tracker, and that 40% real exceeds it. On DanceTrack, behaviorally complex "Multi-group" synthetic data raises Higher Order Tracking Accuracy from 68.8 to 73.5. We will reproduce this recipe on the same splits with the same generator and give the switch point a grid search whose total compute equals our controller's single run plus probe overhead. Our controller beats it if it matches or exceeds its HOTA with less real data or less total compute, or if our single-run forecast reproduces its substitutability curve within the seed-to-seed spread.

For forecasting, the strongest competitor is the oracle that retrains at every mixture ratio and reads off real validation performance. We do not expect to beat its accuracy. We count a win if we come within its seed noise at a fraction of the cost.

The cheap competitor is real-probe validation loss tracked during one synthetic run, using the same $m$ probe examples. This is the key ablation. It has the same data and the same dimension-free concentration, but it ignores gradient direction.

A second ablation replaces the optimizer's own update $u_t$ with the Euclidean gradient direction. This isolates whether optimizer geometry matters.

On tabular data, the reference is the marginal-preserving differentially private synthetic data of Bounding the Excess Risk for Linear Models Trained on Marginal-Preserving, Differentially-Private, Synthetic Data. It reports less than 1% accuracy loss and empirical excess risk below 0.02 at privacy budget $\epsilon=2$ for linear models. Its marginal-matching score serves as the distribution-similarity baseline for predicting how nonlinear models degrade.

Why not just…?

Why not just use classical uniform gradient convergence bounds to certify the synthetic data? Those bounds take a supremum over the whole parameter set. In deep networks they scale as $r^{2l}$ in depth and with the square root of width (Empirical Risk Landscape Analysis for Understanding Deep Neural Networks), so for the tracking and segmentation models here they would demand far more real samples than exist. We will report the bound's numeric value alongside our estimator to make the gap concrete. We do not expect this alternative to compete.

Why not just watch real validation loss on a small probe set during synthetic training? This is the real threat. It is equally cheap and equally dimension-free. If it forecasts the substitutability curve and the switch point as well as $\rho_t$, our contribution shrinks to the theory. We expect it to lag for two reasons. Loss can keep falling while the synthetic update direction already captures only a small fraction of the real descent available. And loss cannot explain why one optimizer benefits from given synthetic data while another does not. The loss-only ablation settles this directly. If it ties, we will say so.

Why not just score synthetic data by distributional similarity, such as matched marginals? For linear models, matching low-order marginals provably bounds excess risk (Bounding the Excess Risk for Linear Models Trained on Marginal-Preserving, Differentially-Private, Synthetic Data). For nonlinear models that guarantee is absent. The tracking results show that photorealism matters less than behavioral structure such as occlusions and crossing trajectories (On the Equivalency, Substitutability, and Flexibility of Synthetic Data). That is evidence that marginal similarity and gradient usefulness can diverge. The tabular experiment tests this by comparing rank correlations with final real accuracy across generators.

Proposed Experiments

The first tier is a controlled setting where real population gradients are exact: Gaussian mixtures and teacher-student networks, as in The landscape of empirical risk for nonconvex losses. "Synthetic" data here comes from a deliberately misspecified generator with tunable mean shift, covariance distortion, and heavy-tailed versus Gaussian noise. We will verify that the estimator's error is flat in width at fixed probe size $m$. We will also check that the predicted stalling level matches the observed real gradient norm, and that the Adam, Muon, and SGD rankings flip as the structure of the bias changes, in the directions the dual norms predict.

The second tier uses the six tabular datasets from the private-synthetic-data paper (Adult, Compas, Churn, Dutch, Law, Heart). We train multilayer networks with each optimizer on synthetic data at $\epsilon=2$. We then ask whether early-trajectory $\rho_t$ ranks generators and optimizers by final real accuracy better than the marginal score and the probe-loss ablation, measured by Spearman correlation across seeds.

The third tier is MOT17 and DanceTrack with the $M^3Act$ generator. Here we forecast the full real-fraction curve from one synthetic run and compare it with the retrained oracle. We also run the controller against the tuned pretrain-then-fine-tune recipe, scored by HOTA and real frames used.

As an optional transfer check, we will ask whether $\rho_t$ computed on a small real probe agrees with the synthetic-tumor validation stream of Synthetic Data as Validation. That paper reports liver-tumor Dice rising from 26.7% to 34.5% in-domain and from 31.1% to 35.4% on FLARE'23.

The decisive checks are three. First, estimator error must stay flat as width grows at fixed $m$. Second, $\rho_t$ must beat probe validation loss in forecasting switch points and real-fraction curves on at least the tabular and tracking tiers. Third, the predicted optimizer-dependence of synthetic data value must appear with the correct sign. Failure on the second check would reduce the proposal to a theoretical account of an existing heuristic, and we would report it as such.