Derivative Agreement as a Test of Whether Synthetic Data Can Replace Real Data

Teams that train on simulated or procedurally generated data need to know, before an expensive training campaign, whether a given generator will actually save them real data. Today the answer comes from two places. One is image-similarity scores, which compare the statistics of synthetic and real samples. The other is a full sweep of pre-train-then-fine-tune runs at several fractions of real data. The first is cheap but often wrong. The second is accurate but costly. We propose a third test, motivated by approximation theory. Train one model on the synthetic data and a small reference model on a little real data. Then compare not only their predictions on real inputs but also their slopes: how each output changes as the input is nudged. Our claim is that agreement in slopes predicts how much real data a synthetic pre-training saves, and that it does so better than agreement in predictions or similarity of the data itself.

The reason comes from a fact that theorists have proved and practitioners rarely use. Two networks can compute nearly the same outputs while their weights sit arbitrarily far apart. The map from weights to functions does not run stably in reverse, which theorists call a failure of "inverse stability". Stability is recovered only when distance between functions also counts derivatives, and only after removing redundant ways of writing the same network. Fine-tuning is a walk through weight space from the synthetic solution to a real-data solution. The length of that walk should therefore be governed by a derivative-aware distance between the two functions, not by an output-only distance. This turns a pure-math result into a measurement for synthetic data.

Core Claim

The approximation-theory side supplies the mechanism. Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size proves that functions realized by a fixed architecture form a set that is not closed. It also proves that function-space convergence can require unbounded weights, so output proximity never guarantees weight proximity. How degenerate is the parametrization of neural networks with the ReLU activation function? identifies three causes of this for shallow ReLU networks:

  • layers rescaled against each other;
  • duplicated neuron directions;
  • neurons that cancel.

Berner et al. then show that inverse stability holds in a restricted parameter space once function distance is measured in a Sobolev norm, one that penalizes mismatch in first derivatives as well as values. Weight distance is then bounded by the square root of that norm. Neither paper says anything about synthetic data.

The synthetic-data side supplies the open problem. On the Equivalency, Substitutability, and Flexibility of Synthetic Data reports three empirical findings:

  • a roughly 30:1 synthetic-to-real frame exchange rate in multi-person tracking;
  • on MOT17, synthetic pre-training plus 20% of real data nearly matches 100% real data;
  • behavioral complexity in the generator, not photorealism, decides whether synthetic data helps. Multi-group scenes lift DanceTrack HOTA (Higher Order Tracking Accuracy) from 68.8 to 73.5, while simple walking data hurts.

The authors offer no account of what sets these numbers. The research file also notes that distributional scores such as Fréchet Inception Distance and Maximum Mean Discrepancy (MMD) do not predict downstream substitutability.

Concretely, let $f_S$ be a model trained only on synthetic data and $g_R$ a reference model trained on a small real set. Both are evaluated on held-out real inputs $x$. We define a two-part gap:

$$D(f_S, g_R) = \underbrace{\mathbb{E}_x \|f_S(x) - g_R(x)\|^2}_{D_0} + \lambda\, \underbrace{\mathbb{E}_{x,v} \|\nabla_x f_S(x)\,v - \nabla_x g_R(x)\,v\|^2}_{D_1}$$

Here $v$ is a sampled input direction and $\lambda$ weights the slope term. $D_0$ is output disagreement and $D_1$ is directional-derivative disagreement. The directional estimator is borrowed from the stochastic version of Sobolev Training for Neural Networks.

The hypothesis is that, across generator variants, $D_1$ ranks generators by real-data savings after fine-tuning. It should do so more faithfully than three alternatives:

  • $D_0$;
  • feature-distribution distances;
  • the zero-shot accuracy of $f_S$ on real data.

Czarnecki et al. use derivatives as a training target and assume an exact derivative oracle, which black-box simulators do not provide. We need no oracle: both functions being compared are trained networks, so their derivatives are always available. Our use is diagnostic, and its object is the generator.

Research Program

The program has three parts.

A controlled setting where the theory nearly applies. We use the low-dimensional regression surfaces from Czarnecki et al. (Styblinski-Tang, Rosenbrock, Ackley and others) and build "simulators" that perturb the true surface in two ways. Some perturbations keep values close while distorting slopes, for example by adding small high-frequency ripples. Others shift values while preserving slopes. We train shallow and moderately deep ReLU networks on each simulator, then fine-tune on a small real sample.

This tests directly whether slope mismatch, not value mismatch, controls the fine-tuning cost. Here we also compute the second quantity of the program: the weight distance between synthetic and real solutions after canonicalization. Canonicalization rebalances layer scales using ReLU's positive homogeneity and merges duplicated or cancelling neurons, following Berner et al.'s degeneracies. We then check whether canonical weight distance tracks $D_1$ as the shallow-network bound predicts, and whether raw weight distance fails to.

This is the deep-network analogue of the parameter-landscape comparison in Is this Model for Real? Simulating Data to Reveal the Proximity of a Model to Reality. That work grid-searched four Bayesian Knowledge Tracing parameters to compare real and simulated data, which is impossible at our scale. Canonicalization is what makes a weight-space comparison meaningful without a grid.

Scaling to real sim-to-real benchmarks. The theory gives no guarantee for deep networks, and the file flags this depth gap as open. The phase diagram of approximation rates for deep neural networks shows that the most efficient deep approximations rely on deliberately discontinuous weight choices. That cuts against us. However, Neural network approximation argues that gradient descent does not reach those bit-extraction configurations in practice. Whether trained deep networks behave like the stable shallow regime is therefore an empirical question, and we treat it as one.

For high-dimensional images, raw input gradients are noisy. We therefore take directional derivatives along data-meaningful directions:

  • interpolation paths between real inputs;
  • standard augmentation directions such as small translations, color shifts and crops, using finite differences;
  • random directions as a control.

We compute $D_1$ at the logits and, separately, at the penultimate features. This tells us whether slope agreement in the representation is what transfers. Measuring on a chosen set of real "measurement points" echoes how Function Space Particle Optimization for Bayesian Neural Networks evaluates particle repulsion on function outputs rather than weights. That paper used the idea to keep ensembles diverse in-domain. We use it to compare a synthetic-trained model against a real one.

Robustness of the reference model. The reference $g_R$ is trained on little data, so its own slopes are unreliable. We use small ensembles of references. We also measure how the ranking degrades as the reference's real set shrinks, against an oracle reference trained on all real data.

Baselines

Naive reference points. The do-nothing reference ranks generators at random, or by synthetic dataset size. The latter is the implicit practice the file describes for large procedural corpora. Ranking by size is directly testable on M³Act, where Chang et al. found that more simple-walking data actively hurt DanceTrack. A useful metric must beat this by a wide margin.

Distribution distances. These compare synthetic and real features: MMD as used in the Deep Adaptation Network, plus covariance alignment distances and Fréchet distance. All are computed in the same frozen feature extractor used for our probes. They are the field's default generator-quality scores. VisDA: The Visual Domain Adaptation Challenge reports that aligning these statistics as a training objective reaches only 51.62% accuracy on VisDA-17 classification, against a challenge-winning 92.8%. That suggests they miss what matters for transfer.

Train-on-synthetic, test-on-real accuracy. This is the zero-shot real-validation performance of $f_S$. It is the strongest simple competitor because it is cheap, uses the same single synthetic run we need, and is what practitioners actually check.

Output agreement alone ($D_0$). This is the closest relative of the proxy-guidance loss in Automated Synthetic-to-Real Generalization, which penalizes KL divergence between a synthetic-trained model's outputs and a frozen real-pretrained network. We use it as an evaluation score rather than a training loss. It is the ablation that isolates our key ingredient: the only difference from $D$ is the slope term.

Fairness and the bar to clear. All scores get the same synthetic training run, the same real reference set, the same held-out real measurement points, and the same tuning budget for any free knob (we tune $\lambda$ and the direction choice on a separate split of generator variants). Beating a baseline means a higher Spearman rank correlation between the score and the measured substitutability. The difference must be confirmed by a bootstrap over generator variants and must hold on at least two of the three benchmarks.

Reported numbers. No reported result was found on Lacuna for the rank correlation of any of these scores with substitutability; establishing those numbers is part of the work. For context on the downstream task, the reported VisDA-17 figures from Chen et al. are 28.2% for plain training on synthetic data, 58.7% with proxy guidance and 61.1% for full ASG.

Why not just…?

Why not just use train-on-synthetic, test-on-real accuracy? This is the real threat, and it could match us. The evidence against it is indirect. Chang et al. show that 100%-synthetic models perform poorly on real data, yet the same synthetic data, used for pre-training, saves up to 80% of real frames. Zero-shot accuracy and fine-tuning value can therefore come apart. Our toy experiment builds simulators that deliberately separate the two, with value-shifted but slope-preserving surfaces predicted to score badly zero-shot yet fine-tune cheaply. On the real benchmarks we report both correlations side by side. If zero-shot accuracy correlates as well as $D_1$ with substitutability on VisDA and M³Act, we will report that slopes add nothing in practice and narrow our claim to the controlled setting.

Why not just compare data statistics with MMD or Fréchet distance? These scores look at inputs, not at what a trained model does with them. The VisDA moment-matching result above suggests the resulting signal is weak. The M³Act finding points the same way: behavioral complexity mattered more than appearance, which is a property of motion structure that pixel-level statistics are poorly placed to see. Both are included as baselines, so the experiment settles the question directly.

Why not just run the full fine-tuning sweep? It is the ground truth, and we run it to obtain the labels. It is also many fine-tuning runs per generator variant across several real-data fractions, repeated whenever the generator changes. Our score costs one synthetic run plus gradient evaluation. If the score does not rank generators nearly as the sweep does, it has no reason to exist, and the experiments are designed to show exactly that.

Why not just add weight decay or proxy guidance and skip diagnosis? Those are training fixes, not evaluations. They do not tell a team which generator configuration to build. Chen et al. also report that plain training on synthetic data peaks early on real validation and then collapses to 28.2%, which makes heuristic stopping unreliable.

Proposed Experiments

Generator variants and ground truth.

  • VisDA-17 classification (CAD renders to COCO). We form about a dozen synthetic variants from subsets defined by viewpoint and lighting ranges and by restricted augmentation. Each is used to pre-train a ResNet-50, which is then fine-tuned on 5%, 10%, 20% and 40% of a real labeled split.
  • M³Act to MOT17 and DanceTrack. We use the generator's own configurations (Dance, WalkRun, Multi-group, and mixtures) with a MOTRv2 tracker, measuring HOTA. Derivatives are taken on the detection and embedding heads.
  • GTA5 to Cityscapes segmentation. This is a third check with FCN-Res50, measuring mean intersection-over-union.

Substitutability is the area under the fine-tuning curve relative to the 100%-real model. As a second ground truth, we also record the real fraction needed to match that model, which is Chang et al.'s quantity.

Decisive checks. There are three, followed by a negative-result criterion.

  1. Toy setting. $D_1$ but not $D_0$ predicts fine-tuning steps and final error across perturbation types. Canonical weight distance tracks $D_1$, while raw weight distance does not. Failure here would falsify the mechanism outright.
  2. Real benchmarks. $D_1$ beats $D_0$, distribution distances and zero-shot real accuracy in rank correlation with substitutability. In particular, it should place M³Act's Multi-group variant above WalkRun for DanceTrack, matching the ordering Chang et al. observed.
  3. Reference size. The ranking from a reference trained on 5% real data stays close to the oracle-reference ranking. Otherwise the score is impractical exactly where it is needed.

If slope agreement only helps at the penultimate features and not the logits, or only along augmentation directions, we will report that as the finding. Such a result would say which derivatives of a synthetic-trained model carry over to real data.