Synthetic data from game engines and CAD renderers is cheap to produce in volume, but models trained on it often fail on real images. Two practical levers are available: train a larger model, or make the synthetic data more varied, with more distinct objects, textures, lighting setups and scenes. Existing evidence tunes each lever on its own. This note proposes measuring how the two interact. Holding the number of synthetic images fixed, we would ask how many distinct synthetic "worlds" a model of a given size must see before its accuracy on real images stops improving. We would then ask whether that number grows or shrinks as the model gets larger, and whether the answer depends on which kind of variation is being added.
The question matters because the two levers cost very different things. A rendering pipeline is expensive to diversify and cheap to run, while a model is expensive to enlarge. If large pretrained models need fewer distinct synthetic worlds, teams should spend their budget on model size and render the same assets many times. If large models instead memorize simulator quirks unless diversity rises with them, the budget should go to asset and scene authoring. Nobody has drawn this curve. Driving in the Matrix found a jump in real-world detection between 10k and 50k GTA V frames and attributed it to environmental variety, but it used one VGG-16 backbone and changed volume and variety together.
Core Claim
The claim is that the diversity threshold for sim-to-real transfer depends on model capacity, and that it depends on capacity differently for different kinds of variation. We hypothesize the following pattern for large pretrained models:
- Unstructured variation needs fewer distinct worlds as models grow. This covers sensor noise, blur and global lighting intensity.
- Structured variation does not. This covers object geometry, surface material and texture, and background composition, where the threshold stays flat or rises with model size.
The hypothesis carries over an asymmetry that Scaling Behavior in Model Fine-tuning for Audio DeepFake Detection reported for speech. Across the Whisper family, scaling model size and synthetic volume steadily improved robustness to additive noise, but gave weak, saturating gains against structured shifts such as pitch shifting and neural codec artifacts. That study treated its synthetic corpus as one undifferentiated pool, so it could not say whether the missing ingredient was diversity. We test whether the asymmetry appears in visual sim-to-real, and whether targeted diversity on the structured axes removes it.
To make "threshold" concrete, we fix the number of synthetic training images $V$ and vary $k$, the number of distinct synthetic worlds those images are drawn from. A "world" is a distinct CAD instance or scene configuration. For each model size $N$ we fit a saturating curve of real-target accuracy:
$$\mathrm{acc}_N(k) = A_N - B_N\,k^{-\gamma_N},$$
Here $A_N$ is the plateau accuracy, $B_N$ is the gap at minimal diversity, and $\gamma_N$ is how fast added worlds pay off. The power-law form follows the saturating scaling fits used for data volume in Scaling Vision Transformers. We define the threshold $k^*_N$ as the smallest $k$ that reaches 90% of the way from $\mathrm{acc}_N(1)$ to $A_N$. The object of study is how $k^*_N$ scales with $N$, estimated separately for each axis of variation.
Research Program
Choice of backbones. We fine-tune one family of pretrained backbones whose members share pretraining data and differ only in size. Candidates are ViT-S, B, L and H from a single pretraining run, or a matched CLIP-style family. Sharing pretraining matters because Data Efficient Neural Scaling Law via Model Reusing shows that large transformers trained from scratch on small data fall into a "curse of data scarcity." In that regime they underperform smaller ones, and the paper places the breakdown below roughly 10% of ImageNet for ViT. That paper limits our design. Synthetic-only training from scratch at VisDA scale would measure trainability failure, not the sim-to-real gap.
The pretraining confound. The confound is real: larger models also absorbed more from real-image pretraining. We do not try to remove it, since that prior is part of what practitioners actually buy when they scale. We do measure it. Each model is also evaluated with a frozen-feature linear probe trained on the same synthetic subsets, which separates what fine-tuning on synthetic data adds from what the pretrained features already carried.
Corpus 1: VisDA-C subsets. The first corpus is the synthetic source of VisDA-C from VisDA: The Visual Domain Adaptation Challenge: about 152k CAD renderings in 12 classes, evaluated on real MS COCO and YouTube Bounding Boxes images. At fixed $V$ we vary the number of distinct CAD instances per class. Where instance identity is not recoverable from metadata, we group renders by near-duplicate clustering. The clustering uses a small embedding model kept separate from the backbones under study, so that "diversity" is not defined by the models being measured.
Corpus 2: controlled re-renders. VisDA-C mixes all axes of variation together, so we add a re-rendered companion set. It uses the same class vocabulary and a standard renderer, with independent controls for:
- geometry, meaning the number of distinct meshes;
- material and texture;
- lighting, meaning the number of illumination setups;
- background;
- sensor effects, meaning noise, blur and exposure.
Each axis is swept while the others are held at a moderate default. This gives one $k^*_N$ curve per axis per model size.
Corpus 3: segmentation. For dense prediction we use GTA5 and SYNTHIA, evaluated on Cityscapes, with diversity at fixed frame count varied in two ways. The first is selecting frames by coverage, using k-center selection in the independent embedding space, versus selecting them from tight clusters. The second is mixing one versus two synthetic sources. The second manipulation connects to Multi-source Domain Adaptation for Semantic Segmentation, which found that naively combining GTA5 and SYNTHIA causes interference between sources. It addressed this with adversarial aggregation on fixed VGG and ResNet backbones and left open whether capacity alone removes the interference. Our grid answers that directly: we check whether the cost of naive multi-source mixing shrinks with $N$.
Baselines
Naive reference. The do-nothing reference is source-only fine-tuning on a random synthetic subset of size $V$, at each model size, with no diversity control and no regularization. On VisDA-C the historical floor for this setting is 28% real-image accuracy for an AlexNet trained on synthetic data, reported in VisDA. Our random-subset runs at each $N$ give the modern equivalent. Every diversity condition must beat its own random-subset run at the same $N$ and $V$ to count.
Strongest method that uses no target data. This is Automated Synthetic-to-Real Generalization (ASG). It adds a KL loss tying the fine-tuned network's outputs on synthetic images to a frozen ImageNet-pretrained copy, and learns layer-wise learning rates with a reinforcement-learning controller. It reports 61.1% on VisDA-C with ResNet-101, against 58.7% for the KL term with fixed learning rates. On GTA5 to Cityscapes it reports 31.47% mean intersection-over-union (mIoU, the standard segmentation overlap score) with FCN-VGG16.
ASG is the right competitor because it attacks the same failure, simulator fine-tuning overwriting real-world features, through the optimizer rather than the data. We run the KL-guidance component at every model size and diversity level, with the same data, steps and a matched hyperparameter search budget. We add the reinforcement-learning schedule at the smallest and largest sizes only, where cost allows. Beating ASG means reaching its accuracy at the same $N$ with diversity selection alone, or showing that diversity and guidance add up.
Reference points that see target images. These are not competitors in our setting, because they use unlabeled real target images:
- VisDA-C: the challenge winner's 92.8% with pseudo-labeling.
- Cityscapes: MADAN's 41.4% mIoU, against 25.8% for naive combined-source training, using the Luodian/MADAN code.
They mark how much of the gap remains for data-only interventions to close.
Key ablations. Two ablations isolate what matters:
- Volume-matched control. Raising diversity while letting $V$ grow confounds diversity with volume, which is exactly the confound in Driving in the Matrix. Every curve is therefore drawn at fixed $V$, and a separate volume sweep is drawn at fixed $k$.
- Per-axis versus pooled diversity. This tests whether a single coverage number predicts transfer, or whether the axis decomposition is needed.
Why not just…?
Why not just scale the model and skip diversity engineering? This is the null hypothesis, and the grid tests it directly. If $k^*_N$ falls toward one on every axis as $N$ grows, the conclusion is that scale substitutes for simulator variety, and we will report that. The audio result in Scaling Behavior in Model Fine-tuning for Audio DeepFake Detection gives a concrete reason to expect otherwise on structured axes, but the visual answer is unknown.
Why not just render more images? Driving in the Matrix shows that more GTA V frames helped, but its gains came packaged with more weather, lighting and pose variety. Our fixed-$k$ volume sweep separates the two effects. If accuracy at fixed $k$ keeps rising with $V$ as steeply as it does with $k$ at fixed $V$, diversity is not the operative variable, and the proposal's practical recommendation collapses.
Why not just regularize toward the pretrained model, as ASG does? This is a real threat. For large backbones, preserving real-image features may matter more than which synthetic images are shown. ASG never tested capacity beyond ResNet-101, so we run its KL guidance across the full grid. If its gain is constant across $N$ and $k$, guidance and diversity are independent levers. If its gain disappears at high diversity or at large $N$, that also determines which lever to use. Either result is informative.
Why not just adapt with unlabeled target images? When real target images are available, adaptation methods are far ahead: 92.8% on VisDA-C against 61.1% for ASG. Our setting is the case where target imagery is unavailable before deployment, or the target is unknown. We report these methods as ceilings, not competitors. We also check whether diversity-selected source data improves adaptation when used as its starting point, since ASG reports that better source training helps later adaptation.
Proposed Experiments and Decisive Checks
Main grid. The core experiment is a grid of four model sizes, five diversity levels per axis, five controlled axes plus the pooled VisDA-C axis, and three seeds. Every run uses fixed $V$ and a fixed step budget. To respect the crossover in which small models win at low compute, as reported in the audio study, we run every condition under two budgets:
- equal steps;
- equal FLOPs (total floating-point operations).
Primary metrics are:
- per-class mean accuracy on VisDA-C validation (real COCO) and test (real YouTube frames);
- mIoU on Cityscapes for the GTA5 and SYNTHIA conditions;
- the fitted $k^*_N$ with bootstrap confidence intervals.
Decisive check 1: the slope of the threshold. The first check is the sign of the slope of $\log k^*_N$ against $\log N$, separately for the unstructured axes (sensor effects, lighting) and the structured axes (geometry, texture, background). The hypothesis predicts a negative slope on the first group and a slope near zero or positive on the second. If both groups slope down together, the asymmetry claim is wrong.
Decisive check 2: diversity versus capacity. The second check is whether the high-diversity condition at the smallest model matches or beats the low-diversity condition at the largest model on real-target accuracy. If it does, simulator variety can substitute for roughly an order of magnitude in parameters, which is the allocation result practitioners need.
Decisive check 3: multi-source interference. The third check is whether naive two-source GTA5 plus SYNTHIA training falls below single-source training at small $N$ and the penalty disappears at large $N$. That would show capacity resolving the source interference MADAN tackled with adversarial discriminators.
References
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Scaling Behavior in Model Fine-tuning for Audio DeepFake Detection
- Scaling Vision Transformers
- Data Efficient Neural Scaling Law via Model Reusing
- VisDA: The Visual Domain Adaptation Challenge
- Multi-source Domain Adaptation for Semantic Segmentation
- Automated Synthetic-to-Real Generalization