Models trained on video-game images, such as the GTA5 driving frames, and tested on real street photos like Cityscapes have improved sharply in recent years. The strongest adaptive system reached 68.3 mIoU (mean intersection-over-union, the standard overlap score for pixel labeling) on GTA5 → Cityscapes, 10.8 points above the previous best. Each such jump, however, ships as a bundle: a new backbone, a new decoder, new pretraining, and several training stabilizers, all introduced together. We do not know which pieces reduce the synthetic-to-real gap and which pieces simply make a better segmenter. A better segmenter would help even on data with no gap at all.
We propose a controlled factorial study that answers this with a simple control. Each component is judged against two reference models: one trained only on synthetic data, and an "oracle" trained on real labels. A component earns credit for sim-to-real transfer only to the extent that it helps the synthetic-trained model more than it helps the oracle. Applied to backbone type, pretraining source, decoder, and three stabilizers, this separates adaptation gains from general capacity gains. It also directly tests a live disagreement over whether transfer comes from large pretrained representations or from task-specific design.
Core Claim
Current attribution in sim-to-real segmentation mixes up two quantities. The first is how much a component raises target accuracy. The second is how much of the gap it closes. A decoder with a wider view of the image may add 3 mIoU on real Cityscapes whether it was trained on GTA5 or on Cityscapes itself. Counting those 3 points as "adaptation" overstates what the field has learned about domain shift.
For a component $c$, let $\Delta_{S\to R}(c)$ be the change in target mIoU from switching $c$ on, for a model trained on synthetic source data (with or without unlabeled-target self-training). Let $\Delta_{R\to R}(c)$ be the same change for a model trained on real labels. We define the gap-specific effect as
$$ G(c) = \Delta_{S\to R}(c) - \Delta_{R\to R}(c). $$
Both effects are averaged over the settings of all other components and over random seeds, following the "marginalized performance" idea of CombinationTS. That paper averages each module's score over a sampled space of evaluation conditions and reports both the mean and its spread, rather than a single best run.
The claim is falsifiable. We predict that a substantial share of the headline gains from recent bundles has $G(c)$ near zero. We also predict that the components with large $G(c)$ are the ones that protect pretrained features from synthetic textures, rather than architectural capacity.
Relation to Prior Work
The closest work is DAFormer. It diagnosed an "architecture gap" in adaptation research and bundled five changes: a Mix Transformer encoder, a multi-level context-aware decoder, Rare Class Sampling, ImageNet Feature Distance, and learning-rate warmup. Its ablations are one-at-a-time and measured only on the adapted target. It never trains the same variants on real labels, so it cannot say whether the decoder closes the gap or just segments better. It also never crosses the stabilizers with the pretraining source. We reuse its codebase (lhoyer/DAFormer) and its training recipe; the new element is the oracle control and the factorial crossing.
VLTSeg motivates the pretraining factor. It fine-tuned CLIP or EVA-02 backbones with a Mask2Former decoder on synthetic data alone and reported a 63.1% domain-generalization mean with no adaptation modules. That paper changes pretraining objective and model size at once, so its result cannot distinguish "language-aligned features" from "a bigger model." A tension from outside vision constrains our expectations. Tan et al. found that removing the pretrained language model from LLM-based forecasters matched or beat the full models (26/26 cases for Time-LLM), and traced the gains to input patching. Jian et al. found that Mamba and Transformer backbones gave no benefit over CNNs for MRI registration, while task-specific logic did. These results warn that fashionable backbones and pretraining are often credited for gains that belong elsewhere. Our design lets the same thing be found, or refuted, for sim-to-real segmentation.
The methodology draws on work that exposed hidden interactions between components. Furuta et al. showed that implementation details in reinforcement learning are tuned to specific algorithms; tanh-squashed policies help SAC but destabilize MPO. We expect similar pairing between feature-distance regularization and ImageNet pretraining. TimeRecipe shows the cost of exhaustive module sweeps (more than 10,000 runs), which is why we use a fractional design.
Research Program
We build one training harness on the DAFormer codebase in which six factors can be switched independently.
The architecture factors are the backbone family (ResNet-101 versus a ViT-B-scale Transformer), the pretraining source (supervised ImageNet versus CLIP), and the decoder (the DeepLabV2 dilated-convolution head versus DAFormer's context-aware fusion decoder). Crossing backbone family with pretraining source is the step that separates "Transformer shape bias" from "pretraining data," the confound left open by both DAFormer and VLTSeg.
The training factors are Rare Class Sampling, Feature Distance, and warmup. Feature Distance is a penalty that keeps selected features close to those of the original pretrained model. It is always computed against the frozen copy of whichever pretrained model initialized the backbone, so its meaning stays constant across pretraining sources.
Each configuration is trained in three regimes. The first is source-only: GTA5 labels, no target images. The second is adaptation: GTA5 labels plus self-training on unlabeled Cityscapes, where the model learns from its own confident predictions, as in DAFormer. The third is oracle: Cityscapes labels. All regimes are evaluated on Cityscapes validation. The source-only regime lets us split $G(c)$ further into a part that shows up without any target data and a part that only appears once self-training is added.
A full crossing is 64 configurations per regime. We run a resolution-V half-fraction of 32 configurations with three seeds each. Under this design all main effects and two-way interactions can be estimated without being confused with one another, provided four-way and higher interactions are small.
DAFormer reports 16 hours per adaptation run on a single RTX 2080 Ti. The 96 adaptation runs therefore cost roughly 1,500 GPU-hours, and the source-only and oracle runs add less than that. We estimate effects and interactions with a linear model over seeds and report the seed-to-seed spread for every effect. This avoids the "best-run" and "high-variance" traps that CombinationTS documents for point-estimate ablations.
A replication pass on SYNTHIA → Cityscapes checks whether the attribution holds for a second simulator. We run only the factors with the largest estimated $G(c)$ there.
Baselines
The naive reference point is the source-only model with a ResNet-101 backbone, ImageNet pretraining, and the DeepLabV2 head, with no stabilizers. It fixes the floor of the gap. For context, VisDA reports 21.4 mIoU for a much older source-only model on GTA5 → Cityscapes, and we expect our floor to be higher. The ceiling is the oracle trained on real labels. The file states only that fully supervised target models exceed 80 mIoU; we will measure our own oracle for every configuration rather than rely on that figure.
The strongest existing adaptive method is the full DAFormer bundle. DAFormer reports 68.3 mIoU on GTA5 → Cityscapes. We rerun it inside our harness with its published hyperparameters and three seeds, so that it becomes one cell of the factorial and not a quoted number. Matching roughly 68 mIoU is a precondition for trusting the harness. The older ProDA, which cleans noisy pseudo-labels using class prototypes and reports 57.5 mIoU, is cited only as historical context.
The strongest no-adaptation competitor is VLTSeg-style fine-tuning of a CLIP backbone on synthetic data alone. VLTSeg reports a 63.1% domain-generalization mean across four real datasets. We reproduce it at matched backbone scale inside the source-only regime; we do not reproduce its 304M-parameter EVA-02 model.
Every cell receives the same iteration budget and the same fixed hyperparameter search of three learning rates, applied identically to all cells. No cell gets hand-tuning.
The ablation that isolates our key ingredient is attribution without the oracle control. This is the conventional $\Delta_{S\to R}(c)$ ranking computed from the same runs. Our proposal "beats" it if the two rankings disagree on at least one component by more than seed noise. That outcome would show that conventional attribution misassigns credit. If the rankings agree everywhere, the oracle control added nothing, and we will report that.
Why not just…?
Why not just run leave-one-out ablations with more seeds? For main effects that is cheaper, and it may prove sufficient. Our design includes that test: if every two-way interaction is within seed noise, leave-one-out would have given the same answer, and we will say so. It still would not include the oracle control, because leave-one-out on the adapted target cannot separate gap closure from capacity. Adding oracle runs to leave-one-out is a legitimate cheaper variant. We will report the attribution it would have produced alongside the factorial one, so readers can see what the interactions add.
Why not just use the largest vision-language backbone and skip attribution? VLTSeg shows that this is a serious threat: large CLIP-style backbones generalize well from synthetic data without adaptation modules. But ToolRL-DR found that scaling up to 32B parameters did not close a sim-to-real gap in a different domain. VLTSeg's own 304M model is also too large for real-time edge use. The pretraining-by-stabilizer interaction answers the question directly. If CLIP pretraining makes all three stabilizers' $G(c)$ vanish, then "just scale" is the right conclusion for this benchmark, and the paper will state it plainly.
Why not just translate synthetic images to look real, pixel by pixel? PixelDA adapts in pixel space with a GAN, a generator trained against a discriminator, and cut LineMod pose error to 23.5°. It notes, though, that this approach addresses color, lighting, and noise rather than geometric shift. It also would not tell us which network components matter. Translation is an input-stage intervention and could be added as a seventh factor. We leave it out of the core design because it would double the compute. We flag it as the first extension if the input stage turns out to dominate.
Decisive Checks
Three outcomes decide the proposal. The first is the decoder. If DAFormer's context-aware decoder has a $\Delta_{S\to R}$ comparable to its $\Delta_{R\to R}$, it is a general segmentation improvement and not an adaptation component. That would revise how its share of the 10.8-point jump is read. The second is the backbone-by-pretraining crossing. If the Transformer advantage in $G(c)$ disappears once CLIP pretraining is applied to both families, the "shape bias" explanation weakens in favor of pretraining data. The third is the Feature Distance-by-pretraining interaction. A large interaction would be a visual counterpart of the co-adaptation Furuta et al. found in reinforcement learning, meaning a stabilizer's value depends on which features it protects. For each of these we report effect sizes with seed intervals, and we repeat the result on SYNTHIA → Cityscapes before claiming it generalizes.
References
- CombinationTS: A Modular Framework for Understanding Time-Series Forecasting Models
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- Strong but Simple: A Baseline for Domain Generalized Dense Perception by CLIP-Based Transfer Learning
- Are Language Models Actually Useful for Time Series Forecasting?
- Mamba? Catch The Hype Or Rethink What Really Helps for Image Registration
- Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
- TimeRecipe: A Time-Series Forecasting Recipe via Benchmarking Module Level Effectiveness
- VisDA: The Visual Domain Adaptation Challenge
- Prototypical Pseudo Label Denoising and Target Structure Learning for Domain Adaptive Semantic Segmentation
- When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks