Fixing What the Simulator Chose to Show Before Fixing How It Looks

Models trained on video-game or simulator imagery lose much of their accuracy on real photographs. The usual explanation is that the renderer gets textures, lighting and sensor noise wrong, and almost all adaptation work attacks that appearance gap. A second and separate cause gets far less attention. The simulator's scripts and level design decide which scenes get generated: how many cyclists appear, how large and far away cars are, where the horizon sits, how often it is dusk. The authors of the GTA-derived datasets say openly that their corpora are badly skewed in this way, for example with road pixels vastly outnumbering bicycles (Playing for Data). In statistical terms this is sample selection bias: the training set was drawn by a mechanism that differs from the one producing deployment data. Simulators make this kind of bias unusually easy to correct. Every synthetic frame comes with perfect labels, so we can compute low-dimensional scene descriptors for it exactly, and on the real side we only need aggregate statistics of those descriptors.

We propose to treat the simulator as a biased sampler over scene content and to remove that bias first. Each synthetic frame is reweighted so that the weighted distribution of exact scene descriptors (class pixel fractions, instance counts, object sizes, camera pitch, time-of-day tags) matches a small set of real-world summary statistics. Only then do we run whatever appearance-level adaptation one prefers. The claim has two parts. Part of the measured sim-to-real gap is a content-selection gap that this cheap, label-space correction closes. That correction also adds to existing adaptation methods rather than duplicating them, because those methods align image features and cannot tell "rendered differently" apart from "sampled differently."

Core Claim and Mechanism

Two strands of selection-bias work supply the method. Fighting selection bias in statistical learning models multiple biased image sources through selection functions. It shows that only an approximate bias estimate is needed for near-optimal risk, and it defines those functions over low-dimensional "anchor" features instead of pixels. In that paper the anchors are hand-picked and the bias is planted artificially on CIFAR. Learning from Biased Data: A Semi-Parametric Approach shows that target summary moments alone, with no individual target samples, are enough to estimate a parametric density ratio. Its reweighted estimator reaches the $O_P(1/\sqrt{n})$ rate one would get by training on target data directly.

Our observation is that a simulator satisfies their assumptions unusually well. The anchors $z$ are computed from ground-truth label maps and engine metadata, not estimated, so they are exact on the synthetic side. Concretely, we fit weights in an exponential-tilting family

$$w_\theta(z) = \exp(\theta^\top \phi(z)) \Big/ \mathbb{E}_{\text{sim}}\left[\exp(\theta^\top \phi(z))\right],$$

where $z$ is a frame's scene descriptor, $\phi$ maps it to a vector of features whose real-world means are known, and $\theta$ is chosen by the generalized method of moments so that

$$\mathbb{E}_{\text{sim}}[w_\theta(z)\,\phi(z)] = \mu_{\text{real}}, \qquad \mu_{\text{real}} = \mathbb{E}_{\text{real}}[\phi(z)].$$

The segmentation or detection network is then trained on synthetic frames with per-frame loss weights $w_\theta$, optionally combined with an adaptation method.

The method deliberately does not estimate density ratios in image or deep-feature space. A Mechanistic Analysis of Sim-and-Real Co-Training reports that linear probes separate real from simulated features almost perfectly. In that regime, feature-space weighting such as Kernel Mean Matching (Correcting Distribution Shift from Sample Selection Bias) will put its weight on whichever few synthetic frames happen to look most realistic, so it measures rendering quality rather than scene content. Label-space anchors have overlapping support across the two domains even when the pixels do not.

The real moments $\mu_{\text{real}}$ come from three sources of decreasing cost. The most expensive is a small labeled real set (10, 50 or 500 images). Cheaper is published dataset-level statistics, such as the class pixel frequencies reported for Cityscapes (The Cityscapes Dataset). The cheapest uses no real labels: a source-trained model's predictions on unlabeled real images, iterated to correct for that model's own confusion. The last option is the riskiest and will be reported separately.

Research Program

The work has three components. The first is the reweighting procedure itself. It requires a descriptor library extracted automatically from GTA5 label maps and from the Sim10k/50k/200k detection sets of Driving in the Matrix. The descriptors are class pixel fractions, per-class instance counts, a log instance-size histogram, the vertical position of the vanishing region, and engine weather and time tags when they are available. Real-side moments of the same descriptors come from Cityscapes or KITTI labels.

The second is a support diagnostic. Selection-bias corrections fail when the source never generates some region of the target, the "support problem" that Fighting selection bias emphasizes. Weights also become unstable when the two distributions diverge, as Robust Importance-Weighted Cross-Validation Under Sample Selection Bias shows. We will therefore report the effective sample size of the weights and the moments that cannot be matched within a weight cap. The diagnostic serves as a constructive output in its own right: a list of real scene configurations the simulator under-produces, which tells the user what to script next.

The third is a gap decomposition. We compare naive synthetic training, content-reweighted synthetic training, and content-reweighted training plus appearance adaptation. Together these estimate how much of the source-to-real drop is content and how much is appearance, a split that current benchmarks blur.

Baselines

The naive reference point is source-only training on unweighted GTA5 with the same backbone. VisDA reports 21.4 mIoU for source-only on its GTA5-to-Cityscapes track and 47.5 mIoU for the winning entry. For detection, Driving in the Matrix reports 0.6803 mAP (Easy) and 0.4207 mAP (Hard) on KITTI for Faster R-CNN trained on Sim 200k. Their published configurations are our reference numbers, and we will reproduce them under our own compute before claiming gains.

The strongest existing competitor is a self-training, pseudo-labeling pipeline of the kind that won VisDA (92.8% on the classification track, 47.5 mIoU on segmentation). It is the strongest competitor because iterative pseudo-labeling on real images can implicitly re-estimate real class proportions, which means it could absorb our effect. We run it with and without our weights, using identical schedules, real unlabeled pools and hyperparameter search budgets. Beating it means a statistically reliable mIoU gain for "self-training + content weights" over "self-training" across three seeds, concentrated on rare classes.

Two simpler reweightings isolate the key ingredient. The first is image-level class-frequency reweighting, matching only class pixel fractions, which is essentially class-balanced training. The second is KMM on penultimate-layer features. If class frequency alone recovers the whole gain, the richer scene anchors add nothing. If KMM matches us, label-space anchors add nothing. Both receive the same real data used to compute our moments.

An oracle ablation subsamples synthetic frames by rejection sampling under the true real-descriptor histogram, using full real labels. It bounds what content correction can achieve and shows how much is lost by using only moments. Hyperparameters for all methods are selected with the same small real validation set. To check that conclusions do not hinge on that set, we additionally use the calibrated synthetic validation protocol of Synthetic data for model selection.

Why not just…?

Why not just use class-balanced losses? Class balancing corrects one marginal. The simulator's skew also covers object scale, count and viewpoint, and these matter most on KITTI's Hard split, where Sim 200k reaches only 0.4207 mAP. This alternative is a real threat. The class-frequency-only ablation decides it, and if it matches the full anchor set we will report that content correction reduces to class balancing.

Why not just rely on self-training, which already adapts to real label statistics? It may partly do so, but its pseudo-labels inherit the source model's bias toward frequent classes, so rare classes stay under-predicted. Our prediction is that the two are additive, with gains on bicycle and rider. If the gain over self-training is under one mIoU point, we will conclude that content selection bias is largely absorbed by modern adaptation and say so.

Why not just generate more or randomized synthetic data? Scaling from Sim 50k to 200k still leaves a large Hard-split gap (Driving in the Matrix). Randomized synthetic weather does not transfer to real adverse conditions: on ACDC, HRNet falls from about 85% to 75.0% mIoU. Randomization also does not tell you which ranges to randomize, a problem Validate on Sim, Detect on Real addresses by needing real validation data. Our support diagnostic is meant to direct new generation toward unmatched content instead of scaling blindly. We test this by spending a fixed rendering budget either uniformly or on the configurations the diagnostic flags.

Why not just translate pixels with PixelDA-style models? PixelDA adapts low-level style while preserving content, so by construction it keeps the content distribution, and its bias, intact. It is therefore a natural partner rather than a rival, and we run it as one of the appearance stages after reweighting.

Proposed Experiments and Decisive Checks

The main experiments run on GTA5 to Cityscapes for segmentation (the VisDA-S protocol), on Sim10k/200k to KITTI for detection, and on GTA5 to ACDC as a stress test. On ACDC, the content anchors include weather and time tags, and the real side is matched to ACDC's condition mix. Metrics are mIoU with per-class IoU, KITTI mAP by difficulty, effective sample size, and the fraction of real moments matched.

The first decisive check is a controlled planted-bias test with known ground truth. We build a deliberately skewed GTA5 subset (for example, cyclists removed from 90% of frames and small cars under-sampled), reweight it toward the full-GTA5 moments, and verify that the method recovers the full-GTA5 model's accuracy. Appearance is identical across the two sets, so any remaining gap must be content.

The second check measures the gain from content weights on top of each adaptation method (source-only, PixelDA, self-training) as the number of real images used for moments varies over 0, 10, 50 and 500. The proposal succeeds if the gains are positive and persist at 50 images. It fails if they vanish under self-training or require hundreds of labeled real images, at which point ordinary fine-tuning would be the better use of those labels. We will run that fine-tuning comparison explicitly at each label budget.