Driving systems are often trained on synthetic data: frames rendered by video-game or graphics engines, where every pixel comes labeled for free. Models trained this way do much worse on real camera images. The usual explanation is that the model learns quirks of one engine, such as its shadows, its edges, or its camera height, and treats them as cues for the objects. Language research already has a general answer to this kind of problem: pool data from several different sources, so that quirks of any one source stop being reliable cues. This proposal tests that answer for synthetic-to-real semantic segmentation, the task of labeling every pixel with a class such as road, car or pole. The setting is strict: no real images are seen during training, and none are used to pick hyperparameters.
There are two claims. First, at a fixed number of rendered frames, splitting the frames across several distinct engines generalizes to real streets better than spending them all on one engine. This should hold even on top of a strong pretrained backbone, and the analysis will measure what the split costs on thin, boundary-heavy classes. Second, the multi-engine setup supplies something the field currently lacks: a way to choose models without peeking at real data. One engine is held out as a stand-in for reality, and the model's score on that engine should predict its real-world score better than ordinary in-source validation does.
Core Claim
The closest existing work already pools simulators, but it needs real target images during training. Multi-Source Domain Adaptation With Collaborative Learning for Semantic Segmentation does the following:
- It trains one network per engine on GTA5, Synscapes and SYNTHIA, with each network softly supervising the others.
- It matches color statistics to Cityscapes.
- It pseudo-labels unlabeled Cityscapes images using the ensemble.
With all three engines it reaches 62.24% mIoU on the 16 classes the engines share. We remove all target access and drop the per-engine networks. We ask whether plain diversity across engines carries the benefit on its own.
The target-free methods work from a single synthetic source:
- Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation restyles GTA5 or SYNTHIA with painting textures. It reaches 43.7% mIoU on GTA5→Cityscapes and never pools engines.
- Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation builds "virtual domains" from GTA5 with heuristic augmentations. It trains leave-one-domain-out prediction heads. We borrow its leave-one-out logic but apply it to genuinely different engines. We also use it for model selection rather than as an architecture.
The budget question itself is unsettled in the file, with evidence pointing both ways:
- Driving in the Matrix finds that scaling GTA V from 10k to 200k frames keeps improving real-world detection.
- PolyGen finds the opposite for image-text pretraining. Spreading a fixed budget across architecturally distinct image generators beats more prompts from one generator.
- PolyGen also reports a slight drop in linear-probe accuracy. That suggests pooling may strip low-level detail, which dense prediction needs.
No one has run the matched-budget comparison for pixel-level tasks. So the claim is falsifiable in two directions: diversity may lose, or it may win overall while hurting poles and signs.
Research Program
We use four engines. GTA5, SYNTHIA and Synscapes are existing datasets. CARLA is a fourth engine we render ourselves, so that its frame count is not a limit. All labels are mapped to the 16 classes the engines share with Cityscapes. Classes an engine lacks are marked "ignore" rather than "background." This follows the negative-transfer lesson from Augmenting Biomedical Named Entity Recognition with General-domain Resources: there, general-domain text left "patient" unlabeled while the target corpus labeled it, which hurt recall.
The study has three parts.
Budget sweep. For several total budgets $B$, we compare three ways to spend the frames:
- $B$ frames from a single engine, done for each engine.
- $B$ split equally across all four engines.
- Every split that uses two or three engines.
$B$ is capped by the smallest engine's size so that every split is feasible. Model, schedule and augmentation are identical across arms. Each arm is trained on two backbones: an ImageNet-pretrained ResNet-101, and a frozen-then-lightly-tuned DinoV2-style vision transformer. DinoV2 is one of the foundation encoders that Domain Generalization for Semantic Segmentation: A Survey credits with the field's large jump.
What pooling removes. We report per-class IoU and boundary F-score, with thin classes (pole, sign, light, person) singled out. We also measure how much the trained features rely on high-frequency content, by checking accuracy under low-pass filtering of the input. This tests the tension PolyGen raises.
Held-out-engine selection. We generate a pool of candidate configurations, varying the following:
- learning rates and checkpoints;
- texture randomization on or off;
- engine mixtures.
For each candidate we compute three proxies:
- its score on in-source validation frames;
- its leave-one-engine-out score, meaning it is trained on three engines and evaluated on the fourth, averaged over rotations;
- prediction entropy on unlabeled synthetic frames.
We then measure how well each proxy ranks the candidates against their true real-world mIoU.
Baselines
Naive single-engine reference. A model trained only on GTA5 with standard augmentation, at the same budget and backbone. VisDA reports 21.4 mIoU for a GTA5 source-only model on Cityscapes in VisDA: The Visual Domain Adaptation Challenge, with an older network. Our reference is rerun on our backbones, so that every comparison is matched.
Single-source texture randomization. GTR+LTR+CGL from Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation is the strongest target-free method in the file that works on classic backbones, at 43.7% mIoU on GTA5→Cityscapes. We rerun it on GTA5 at each budget with our ResNet-101 and the same tuning budget. Beating it means the equal four-engine split, with no painting styles, exceeds it on Cityscapes and on the other real targets. Because the two are not exclusive, we also run randomization applied to the pooled data.
Foundation-model domain generalization. The survey reports SoRA at 68.3% and MFuser above 71% mIoU on GTA5→Cityscapes. These are the strongest numbers overall. We do not claim to beat them with a weaker backbone. The comparison that matters is within the foundation backbone: does the four-engine split add a significant gain over GTA5-only at matched budget and matched fine-tuning steps? No reported result was found on Lacuna for foundation-model generalization trained on pooled multiple simulators.
Target-accessing upper reference. Co-Learning's 62.24% mIoU on 16 classes, with all three engines and unlabeled Cityscapes, marks how much target access is worth. It is a ceiling for context, not a competitor.
Ablations. Two ablations isolate the key ingredient.
- Diversity versus selection. All four engines' frames are used, but checkpoints are chosen by in-source validation instead of the held-out engine.
- Sources versus taxonomy fixes. Two engines are pooled with the ignore-label fix switched off.
Why not just…?
Why not just use a foundation model, since it already reaches about 70% from GTA5 alone? This is the most serious threat. Real-world pretraining may already erase the engine quirks that pooling targets, and in that case the budget effect would vanish on DinoV2. The survey also notes the "proxy data paradox": these backbones have already seen billions of real images. The experiment settles the question directly. The budget sweep is run on both backbones. If the multi-engine gain shrinks to noise on the foundation backbone, we report that claim one holds only for weaker pretraining. Claim two, target-free model selection, matters regardless, because foundation-model methods are also tuned somewhere.
Why not just apply heavy style randomization to one engine? Restyling changes texture but not geometry, camera rig, object layout or label conventions. Engines differ in exactly those things. Still, randomization may capture most of the gain. The pooled-plus-randomization arm and the randomization-only arm, at matched budget, will measure how much of the multi-engine effect is texture alone.
Why not just render more frames from the best engine? Driving in the Matrix suggests single-engine scaling had not saturated at 200k frames. Our sweep compares the scaling curves directly. If the single-engine curve crosses above the pooled curve at larger budgets, that is a finding, and we report the crossover budget.
Why not just concatenate the engines without any of this? We do. Naive concatenation is our main treatment, not a strawman. What we add is not a training trick. It is the matched-budget evidence, the per-class cost analysis, and a validated selection rule.
Why not tune on Cityscapes validation as most papers do? That leaks target information into a supposedly target-free method. Measuring how much selection by held-out engine loses relative to that oracle quantifies the leak.
Proposed Experiments and Decisive Checks
Every model is evaluated on three real datasets that Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation also uses: Cityscapes validation, BDDS and Mapillary. We report mIoU over 16 classes with three seeds. No reported result was found on Lacuna for SYNTHIA-only numbers on these targets, so all single-engine numbers are measured by us.
There are three decisive checks.
- Budget. The four-engine split must beat the best single engine at the same budget on at least two of three real targets. The margin must exceed seed variance. This must hold on the ResNet backbone, and is tested separately on DinoV2.
- Thin classes. Pooling must not reduce boundary F-score on pole, sign and light by more than its overall mIoU gain. If it does, the PolyGen-style loss of detail carries over to dense prediction. We would then test whether adding the engines' free contour labels as an auxiliary task restores it. That idea comes from Cross-Domain Self-Supervised Multi-Task Feature Learning Using Synthetic Imagery.
- Selection. Across the candidate pool, the held-out-engine score must rank candidates by real-world mIoU significantly better than in-source validation does. It must also close most of the gap to choosing by Cityscapes validation. If the held-out engine is too easy, its scores will not spread out enough to rank candidates well. That would falsify claim two, and it is reported as such.
References
- Multi-Source Domain Adaptation With Collaborative Learning for Semantic Segmentation
- Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation
- Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensembles
- Augmenting biomedical named entity recognition with general-domain resources
- Domain Generalization for Semantic Segmentation: A Survey
- VisDA: The Visual Domain Adaptation Challenge
- Cross-Domain Self-Supervised Multi-Task Feature Learning Using Synthetic Imagery