Layout-Blind Global Critics for Sim-to-Real Image Translation

A common way to use simulator data is to "repaint" synthetic images so they look real, then train a perception model on the repainted images using the simulator's free, exact labels. The repainting is usually done by a generative adversarial network (GAN), a generator trained against a critic that tries to tell its outputs from real photos. The critic's field of view is the crux. If it sees the whole image, it notices that synthetic scenes contain objects shaped differently from real ones, and it pushes the generator to reshape them, which silently invalidates the labels. If it sees only small patches, shapes survive, but nothing enforces that lighting, shadows and color are consistent across the image. Jaipuria et al. measured this trade-off directly: shrinking the critic's receptive field from 46×46 to 16×16 or 10×10 pixels preserved object shape, while 7×7 lost photorealistic depth and illumination. The receptive field size is currently picked by hand.

We propose a different split. One part of the critic stays small and looks at raw pixels for texture and sensor noise. The other part looks at the whole image but is structurally prevented from judging scene layout. It receives the scene's semantic layout as a fixed input that the generator cannot change, and its layout pathway is trained so that it carries no information about which domain the layout came from. This global critic can still reward image-wide appearance that is plausible given a layout, such as shadows that fall consistently or sky brightness that matches the ground. It cannot reward changing the layout, because it never learned which layouts are "real." The question we test is whether such a critic removes the hand-tuned realism-versus-retention trade-off and produces translated synthetic data that measurably helps a strong modern adaptation pipeline.

Core Claim

Long-range context and small-patch discrimination stop conflicting once the global critic is denied the one cue that causes geometry warping: the domain identity of the layout. Concretely, the discriminator has two branches. The local branch is a PatchGAN-style convolutional critic, of the kind popularized by Image-to-Image Translation with Conditional Adversarial Networks and used for sim-to-real refinement in SimGAN, with a receptive field of about 16×16. The global branch takes that branch's patch features, concatenates an embedding of a semantic layout map $s$, and mixes information across the whole image with recurrent criss-cross attention from CCNet. Criss-cross attention lets each position attend along its row and column, and stacking two such steps reaches every pixel at $O(N\sqrt{N})$ cost, which keeps full-resolution Cityscapes frames tractable. Translated synthetic images are paired with their exact simulator labels. Real images are paired with pseudo-label layouts from a segmentation network.

The key ingredient is a small auxiliary head $c$ that tries to predict the domain from the layout embedding $e(s)$ alone, trained through a gradient-reversal layer so that $e$ is pushed to make domain unpredictable. The discriminator objective is

$$ \mathcal{L}_D = \mathcal{L}_{\text{adv}}\big(D_{\text{loc}}(x)\big) + \lambda\,\mathcal{L}_{\text{adv}}\big(D_{\text{glob}}(x, e(s))\big) - \mu\,\mathcal{L}_{\text{CE}}\big(c(e(s)), d\big), $$

where $x$ is an image, $d$ its domain label, $\mathcal{L}_{\text{adv}}$ the usual real-versus-fake loss, $\mathcal{L}_{\text{CE}}$ cross-entropy, and $\lambda, \mu$ weights. With the minus sign, the discriminator is rewarded when its layout pathway gives the auxiliary head no domain information. The generator gets gradients only through $x$. Because $s$ is fixed and $e(s)$ carries no domain information, the global branch has no learned preference for real-world shapes and therefore no reason to bend synthetic ones.

Research Program

The work proceeds in three stages. First, we build the translator: a self-regularized refiner as in SimGAN, trained against the two-branch critic, on GTA5 (Playing for Data) to Cityscapes (Cityscapes). Real-image layouts come from a DAFormer model trained on raw GTA5, run through the official code. Second, we analyze what the global branch actually uses. We feed it images whose layout has been swapped with that of another image and check that its score tracks appearance-to-layout consistency rather than layout realism. We also check that the auxiliary domain head stays at chance. Third, we measure downstream value by retraining DAFormer on translated GTA5 and evaluating on Cityscapes.

Pseudo-labels on real images make the global branch depend on a segmenter trained on the very gap we are closing. We report sensitivity to this by repeating the run with layouts from a weaker, source-only segmenter. A smaller secondary study on UnityEyes to MPIIGaze, using coarse eye-region layouts, tests whether the mechanism transfers to a non-driving task where SimGAN is the reference.

Baselines

The naive reference point is DAFormer trained on unmodified GTA5, as in DAFormer. This is also the strongest existing method on the question, because it already closes much of the gap through a Transformer backbone and self-training. It reports 68.3 mIoU on GTA5 to Cityscapes. We will rerun it from the official code with three seeds on the same hardware class. Only the training images change between this run and ours: raw versus translated GTA5. Schedule, stabilizers and tuning budget stay identical. Beating it means a mean gain over our reproduced number that exceeds the seed-to-seed spread, with the gain concentrated in illumination-sensitive classes rather than in a single class.

The second baseline is a SimGAN-style patch-only critic, with the receptive field swept over 7, 10, 16 and 46 pixels. This reproduces the Jaipuria et al. trade-off on our data. We give it the best receptive field chosen on a held-out validation split, which is more tuning than our method receives.

The third baseline is a multi-scale critic: patch plus an unconditioned whole-image branch with the same criss-cross attention and parameter count. It isolates whether global context alone helps or whether it reintroduces warping.

The key ablation is our critic with layout conditioning but without the gradient-reversal domain head. This shows whether removing domain information from the layout pathway is what prevents warping.

For feature-level reference points we cite the reported numbers rather than rerun them. CDANet (Context-Aware Domain Adaptation) reports 49.2% mIoU with global cross-domain attention, and patch-mode alignment (Discriminative Patch Representations) reports 41.3% alone and 46.5% with CyCADA. On gaze, SimGAN reports 7.8° error against 11.2° for raw synthetic data.

Why not just…?

Why not just pick the best patch receptive field and stop? This is the most honest threat. A tuned 16×16 critic may already give most of the benefit. The receptive-field sweep in our baselines answers it directly. If no setting of the patch-only critic matches our method on both geometry retention and downstream mIoU at once, the trade-off is real and our critic removes it. If the 16×16 critic ties us, we will report that the global branch adds nothing measurable on this benchmark.

Why not just add an ordinary global discriminator next to the patch one? The evidence predicts it will warp shapes. Jaipuria et al. found that large-field critics reshaped synthetic A-frame couplers into straight ones. GL-MSDA reports that CycleGAN translation lowered grasp AP from 33.93 to 32.7, below the unadapted model. Our multi-scale baseline tests this on driving scenes, and the label-consistency metric below will show whether warping occurs.

Why not just self-train a Transformer and skip translation entirely? DAFormer shows this is very strong, and translation may be redundant on top of it. We take this seriously: the decisive comparison is DAFormer on raw versus translated GTA5 under an identical recipe. A null result there would be a legitimate finding that pixel-level adaptation is obsolete for this benchmark. In that case we would report the geometry-retention results only as a contribution to translation itself.

Why not just randomize the simulator? Chen et al. show that randomization only covers real conditions that fall inside the randomized range. GTA5 renders are fixed, so we cannot test randomization here. We note this as a limitation of the benchmark rather than claiming to refute it.

Proposed Experiments

The first decisive check is geometry retention. Following Jaipuria et al., we compute the Modified Hausdorff distance between edge maps of the source and translated images. We also compute label consistency: the mIoU between the GTA5 ground truth and the output of a Cityscapes-supervised segmenter applied to the translated image. If translation warped geometry, this agreement drops. Our critic should match the 10×10 patch critic on both measures, while the unconditioned multi-scale critic should be worse.

The second check is realism. We report Fréchet Inception Distance (FID) to Cityscapes, computed globally and on large crops. Our critic should match the 46×46 critic on FID while keeping the retention of the small-field critics.

The third check is downstream value: Cityscapes validation mIoU for DAFormer trained on each translated set, three seeds each, with per-class breakdowns. We also add SYNTHIA to Cityscapes, where DAFormer reports a 5.4 mIoU gain over prior work; we will use our own reproduction as the reference there.

The fourth check probes the mechanism. With layout-swap inputs, the full model's global score should drop when appearance contradicts the layout and stay flat when the layout alone changes domain. The no-reversal ablation should not show this pattern.

The secondary gaze study compares our critic to SimGAN's reported 7.8° error.