Models trained only on rendered images tend to fail on real photographs. A common remedy starts from a network pre-trained on real photos (ImageNet) and fine-tunes it on synthetic data. Fine-tuning then overwrites much of what the network knew about real images, and the network learns rendering artifacts in their place. We propose to keep two representations of every input instead: one from a frozen copy of the real-photo network, and one from its synthetic-fine-tuned counterpart. A small gate mixes the two, dimension by dimension. The gate does not look at the features themselves. It looks at a cheap measure of how much this particular input looks like it came from the simulator, and trusts the real-photo representation more as the input moves away from simulator statistics.
This design comes from speaker recognition. In that field, fusion that is steered by an explicit estimate of the recording condition carries over to unseen noise conditions better than fusion weights fitted on the training condition. The central difficulty is one the speech work never faced. In sim-to-real, the training set contains no real inputs by construction. A gate fit on clean synthetic data will learn to trust the synthetic branch everywhere, because on that data the synthetic branch is always better. The proposal is built around breaking this circularity, and the experiments are designed to show whether that works.
Core Claim
Instance-level, condition-driven fusion of a frozen real-prior embedding with a synthetic-adapted embedding beats three alternatives on synthetic-to-real domain generalization, where no real images are used in training: output-level distillation from the frozen model, fixed-weight mixing of the two models, and single-model augmentation. It does so only if two conditions hold. First, the gate sees a style descriptor of the input rather than its semantic features. Second, the gate is trained on synthetic inputs that have been deliberately pushed off the simulator's statistics, so it learns the relation "far from simulator → trust the real prior." If this relation does not transfer to real images, the method reduces to static fusion, and the experiments below are built to detect that outcome.
Where This Sits
Fusion lineage. The fusion mechanism comes from multi-embedding work in speech. Combination of deep speaker embeddings for diarisation fuses embeddings from architecturally different extractors using attention, gating and bilinear pooling. It reports 29% and 15% relative speaker-error reductions on AMI and NIST RT05 over the best single system, and finds that fusion beats scaling one network. That work fuses embeddings that were all trained on the same kind of real data. We fuse embeddings that differ in which domain shaped them.
The condition-driven gate comes from A noise-robust system for NIST 2012 speaker recognition evaluation. That system uses an explicit prediction of the acoustic environment to reweight its subsystems (C_primary 0.183). It never faced a shift that was absent from training. Our setting is exactly that case.
Methods we compete with. Automated Synthetic-to-Real Generalization (ASG) motivates the proposal and is its main competitor. It also keeps a frozen ImageNet model, but couples the two models only through a KL loss on class probabilities, plus a reinforcement-learned controller for per-layer learning rates. At test time only the student remains, so any real-image knowledge must survive inside one set of weights. We keep both representations at inference and decide per input how much each contributes.
Breaking the Synthetic-Real Domain Shortcut (DREAM) supports the premise that synthetic artifacts occupy a shared low-dimensional feature subspace. It removes that subspace with one global projection. Our gate is instance-conditional instead, so it can leave features intact on inputs where no artifact is present.
Source of the gating signal. Domain2Vec supplies the descriptor. It represents a domain by Gram-matrix style statistics of convolutional activations, and reports that distance between such embeddings correlates with cross-domain accuracy (Pearson −0.774). That is evidence at the dataset level that style distance predicts transfer failure. We test whether the same holds per instance and use it to drive fusion.
Source of the shift used to train the gate. PASTA reports that synthetic images lack the high-frequency amplitude variance of real ones. Its amplitude perturbation is our tool for manufacturing "off-simulator" inputs. It is also our strongest augmentation baseline.
Research Program
Model. Let $e_r(x)$ be the embedding from the frozen real-pretrained backbone. Let $e_s(x)$ be the embedding from the same architecture after fine-tuning on synthetic data. Let $s(x)$ be a low-dimensional style descriptor: downsampled Gram statistics of the frozen network's first two stages, which is cheap because those stages are shared. The fused embedding is
$$z(x) = g(s(x)) \odot e_r(x) + \big(1 - g(s(x))\big) \odot e_s(x),$$
where $g$ is a small network outputting per-dimension weights in $[0,1]$ and $\odot$ is elementwise product. A task head reads $z$. For segmentation the same form is applied to spatial feature maps, with $s$ computed per region.
Training in three stages. First, fine-tune $e_s$ on synthetic data with standard augmentation. Second, build a gate-training set from held-out synthetic images. These are passed through PASTA amplitude perturbation and random channel-statistic shifts of varying strength, so they span a range of distances from the simulator's style. Third, train $g$ and the head on this set with both backbones frozen.
On clean held-out images, the synthetic branch should win. On strongly shifted images, the frozen branch should win. The gate is only ever shown the style descriptor, so it can express "how simulator-like is this?" but cannot learn semantic shortcuts. A light calibration term ties the average gate value to the observed per-bin loss gap between the two branches.
Analysis. Beyond accuracy, we measure whether the gate behaves as intended on real images. We compute per-instance correlation between $g$ and which branch alone classifies the instance correctly. We also check whether gate values on real images fall outside the range seen during gate training, which would mean the relation had to extrapolate rather than interpolate.
Baselines
Source-only reference. The naive point is the synthetic-fine-tuned model alone, that is, $e_s$ with its head. ASG reports 28.2% on VisDA-17 for this kind of model trained with a large learning rate. Its companion is the frozen real-prior backbone with a linear head trained on synthetic data, which tells us how much real knowledge is usable without fine-tuning.
Strongest method on classification: ASG. It shares our ingredient of a frozen real-photo model and is evaluated in the same no-real-data setting. It reports 61.1% on VisDA-17 (58.7% with proxy guidance alone) and raises FCN-Res50 on GTAV→Cityscapes from 22.17% to 29.64% mIoU (ASG). We rerun proxy guidance with the same backbone, data and tuning budget. We also cite the full reinforcement-learning result as published, because reproducing the controller is expensive; this is a stated asymmetry.
Strongest method on segmentation: PASTA. It is the strongest reported syn-to-real number in the file, 28.95→44.12 mIoU on GTAV→Cityscapes with DeepLabv3+ (PASTA). We run it on the same backbone we use. Because our gate training already uses PASTA-style perturbations, the fair comparison is our method against the synthetic branch trained with PASTA, not against an unaugmented model.
Fixed-weight mixing. This is the most dangerous baseline. We mix $e_r$ and $e_s$, or their logits, with a single scalar weight, reported two ways: with the weight chosen on held-out synthetic data, which is the fair version, and with the weight chosen on real validation data, an oracle upper bound for any static fusion.
Capacity-matched ensemble. Two synthetic-fine-tuned models from different seeds are ensembled. This separates "benefit of the real prior" from "benefit of running two networks," echoing the fusion-beats-scaling finding of Sun et al..
DREAM-style projection. A global SVD projection of $e_s$, adapted from DREAM by using clean-versus-perturbed synthetic feature differences in place of real ones.
Ablations isolating the key ingredients. One ablation feeds the gate $[e_r, e_s]$ instead of $s(x)$. Another trains the gate on clean held-out synthetic data instead of perturbed data.
What counts as success. Beating the method means exceeding the fair fixed-weight baseline and the rerun ASG on VisDA-17, and exceeding PASTA-alone on GTAV→Cityscapes. In each case the margin must exceed seed variance, measured over three seeds.
Why not just…?
Why not just mix the two models with one fixed weight? This is the real threat. If the benefit comes entirely from having the real prior available at inference, a single scalar captures it, and the gate is decoration. The experiments settle it directly. If our method does not beat the fair fixed-weight baseline, we will report the fixed-weight mix as the finding and drop the gate. If it beats the fair version but not the real-tuned oracle, the gate is at least recovering a weight that could not otherwise be chosen without real data. That is a narrower but honest contribution.
Why not just use ASG's distillation loss? ASG forces the real-image knowledge through one set of weights, and its 61.1% needs a costly learned controller (ASG). Our comparison against rerun proxy guidance on the identical backbone shows whether keeping both representations at inference beats squeezing them into one. The test-time cost of the second backbone is reported alongside, since ASG pays none.
Why not just augment with PASTA? PASTA is cheap and strong: +15 mIoU on GTAV→Cityscapes and +16.9 mAP on Sim10K (PASTA). It is complementary rather than competing, and we stack on top of it. If PASTA-trained $e_s$ alone matches our fused model, then augmentation already closes the gap that the real prior would fill, and the proposal fails on segmentation.
Why not just project out the synthetic subspace, as DREAM does? DREAM assumes a single domain-bias subspace shared across inputs and removes it everywhere (DREAM). It was also built with access to real features, which we do not have. The adapted projection baseline tests whether a global linear correction suffices. The gate is only justified if it does better on inputs whose style descriptor says they are simulator-like, where projection needlessly removes variance.
Proposed Experiments
Classification. VisDA-17 is the primary benchmark (VisDA). We train on the 152k CAD renderings, use no real data, and evaluate on the COCO validation images. The backbone is the ResNet used by ASG.
Segmentation. GTAV→Cityscapes mIoU with DeepLabv3+ is the secondary benchmark, reported alongside the 44.12 PASTA figure. BDD100K and Mapillary serve as additional real targets. We found no reported result on Lacuna for PASTA on those two.
Detection, conditional. Sim10K→Cityscapes mAP is run only if segmentation succeeds.
Decisive checks. Three results decide the proposal. The first is the fair-fixed-weight comparison described above. The second is the per-instance diagnostic on real images: if gate values do not correlate with which branch is correct, any accuracy gain is coming from something other than the claimed mechanism. The third is the gate-trained-on-clean ablation, which should collapse toward the synthetic branch and lose most of the gain; if it does not, the perturbation-based gate training is not what matters.
Failure mode to probe. A final check targets the most likely way the method breaks. We bin real test images by style-descriptor distance and ask whether real images sit beyond the range of the perturbed synthetic set. If they do, the gate is extrapolating. We would then report whether widening the perturbation range repairs this or whether the approach is limited by how well the manufactured shift imitates real shift.
References
- Combination of deep speaker embeddings for diarisation
- A noise-robust system for NIST 2012 speaker recognition evaluation
- Automated Synthetic-to-Real Generalization
- Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
- Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation
- Pasta: Proportional Amplitude Spectrum Training Augmentation for Syn-to-Real Domain Generalization
- VisDA: The Visual Domain Adaptation Challenge