Synthetic driving datasets such as GTA5 come with labels that are exact at the pixel level, because the labels are pulled straight from the game engine's rendering calls. Models trained on these images still fail on real photographs, so a common fix is to repaint the synthetic images to look real before training. That repainting quietly damages what made synthetic data valuable. Convolutional image translators smear color and texture across object outlines, a defect known in colorization as color bleeding. They also blur thin objects such as poles and bicycle frames, so the repainted image no longer lines up with its own perfect label mask. The two existing answers sit at opposite extremes. One lets the generator move edges and hallucinate objects freely. The other strips out all spatial processing so that only per-pixel color can change.
We propose a translator that sits between these extremes and is enforced by construction rather than by a loss. Every internal operation of the generator may gather information only from pixels that share the same synthetic class label, and the generator runs at full image resolution throughout. Cross-boundary leakage is therefore impossible, while texture, noise and lighting can still be translated richly inside each region. The one legitimate kind of cross-boundary mixing, optical blur from a real camera, is reintroduced through a separate tiny lens-blur model that does not depend on labels. In two sentences: synthetic images already tell us exactly where every boundary is, so the translator should be physically unable to push color or texture across those boundaries. Real-camera blur, which does cross boundaries, should be modeled as an explicit low-capacity physical step instead of being learned as uncontrolled semantic leakage.
Core Claim
Sim-to-real image translation loses accuracy at boundaries and thin structures for two separable reasons. The first is semantic leakage, where content from one class spreads into another. The second is resolution loss inside the generator's downsampling bottleneck. Both can be removed without giving up the ability to change texture. The test is whether a label-gated, high-resolution translator used as the data-preparation stage of an otherwise fixed adaptation pipeline produces three outcomes. It should yield measurably higher accuracy in narrow bands around object boundaries and on thin classes than a CycleGAN-style translator. It should also beat a per-pixel color mapper, and should not lose accuracy on broad classes. A deliberately chosen outcome could falsify the idea: a per-segment color-only mapper may match the full method. That result would mean texture translation contributes nothing once leakage is controlled.
Relation to Prior Work
Three works motivate the problem. CyCADA combines CycleGAN pixel translation with feature alignment. It adds a semantic consistency loss that asks a pretrained source segmenter to give the same labels before and after translation. That loss is soft and is measured through a downsampled segmentation head, so it cannot forbid sub-object smearing along silhouettes. ColorMapGAN shows the failure of unconstrained translators directly. On Luxcarta satellite imagery (Béziers to Roanne), CycleGAN reached 14.22% overall IoU, against 42.88% for ColorMapGAN's per-pixel affine color generator. That generator cannot alter texture, sensor noise or lighting structure, which are the dominant gaps in rendered driving scenes. The LAB statistics matching in Multi-Source Domain Adaptation With Collaborative Learning is even more conservative: one global shift per image, worth about 3.8% mIoU over direct transfer. Our design generalizes both color mappers. Its receptive field is large, but it is confined to a labeled region.
Two works supply the method. From colorization, DDColor separates a full-resolution pixel pathway from a semantic color pathway. Deep Edge-Aware Interactive Colorization finds that correcting bleeding early in the encoder works better than correcting it in the decoder, and that PSNR is blind to bleeding. Both lead us to constrain every layer rather than post-process the output, and to build a dedicated leakage metric. The edge-aware work needs human scribbles at inference time to locate boundaries. In our setting those boundaries come free from the engine (Playing for Data, SYNTHIA). From recognition, HRNet keeps a full-resolution stream for the whole network depth while exchanging information with lower-resolution streams. This preserves thin structures. HRNet has never been used inside a sim-to-real translator, and has only been evaluated with in-domain supervision.
One paper constrains the proposal. MRFP shows that randomly perturbing high-frequency features and channel statistics during training already improves generalization from synthetic data without any translation: ResNet-50 rises from 31.21% to 39.27% mIoU averaged over five real targets. This is the main threat to the premise that translation is worth doing, and our experiments test against it head on.
Research Program
The translator processes a synthetic image $x$ together with its label map $y$, and is trained adversarially against unlabeled real images. Its basic operation is a label-gated convolution, which renormalizes the convolution weights over neighbours that share the centre pixel's label:
$$h'(p) = \frac{\sum_{q \in N(p)} \mathbb{1}[y(q)=y(p)]\, W_{q-p}\, h(q)}{\sum_{q \in N(p)} \mathbb{1}[y(q)=y(p)]}$$
Here $h$ is a feature map, $p$ is a pixel, $N(p)$ is its kernel neighbourhood, and $W_{q-p}$ is the kernel weight for the offset from $p$ to $q$. This is the partial-convolution idea with the validity mask replaced by "same label as the centre."
Long-range context follows the HRNet layout, with one change. The low-resolution streams are not strided feature maps, which would mix classes. Instead they are per-segment tokens, formed by pooling high-resolution features over each connected labeled region. The tokens interact through attention and are broadcast back only to pixels of their own segment. A region can therefore learn that "this is a whole road under overcast light" and add matching texture, while the full-resolution stream never downsamples.
A final shared, label-independent blur kernel with a handful of parameters models the camera's point-spread function, applied after a learned per-pixel noise model. This lets the translator imitate real-camera softness at edges without opening a semantic channel. We use GTA5 class labels, so leakage between adjacent instances of the same class remains possible; we report it separately.
The translated images feed a fixed downstream pipeline: an HRNetV2-W48 segmenter, plus one fixed self-training recipe for the adaptation setting. Only the translator changes between conditions. Alongside it we build two analysis tools. The first is a translation-leakage score: the chroma and texture correlation between the two sides of a labeled boundary in translated images, minus the same quantity in the source image, adapted from the cluster-discrepancy idea in the edge-aware colorization work. The second is a boundary-band evaluation of segmentation, which computes IoU only within $k$ pixels of ground-truth boundaries in Cityscapes, for $k$ from 2 to 16.
Baselines
The naive reference point is source-only training of the same HRNet segmenter on raw GTA5, with no translation. Every gain is measured against it. A cheap reference point is global LAB mean and standard deviation matching, as in the collaborative multi-source work, which reported roughly 3.8% mIoU over direct transfer in its own setup.
The strongest translation competitor on this exact question is CyCADA-style CycleGAN translation with its semantic consistency loss. It is the established way to translate texture while discouraging label flipping, and it reports 83.6% per-pixel accuracy on GTA5 to Cityscapes against a 54.0% source-only baseline (CyCADA). We re-run it with our segmenter rather than quoting it, because its numbers use an older backbone and per-pixel accuracy. The second translation competitor is ColorMapGAN applied to driving scenes.
Two more baselines check the end-to-end comparison. The first is MRFP+ applied to the HRNet segmenter on raw GTA5; it reports 39.27% averaged mIoU with ResNet-50 under domain generalization. The second is the strongest single-source adaptation number in the file, 60.2% mIoU on GTA5 to Cityscapes and 56.5% on SYNTHIA to Cityscapes, from Unsupervised Contrastive Domain Adaptation. We treat the latter as a reference ceiling for the full pipeline, not a like-for-like comparison. No reported result was found on Lacuna for boundary-band IoU or thin-class IoU under any of these methods, so all boundary numbers will be produced by our own runs.
Fairness rules apply to every translator. Each gets the same synthetic images, the same unlabeled real images, the same training budget in GPU-hours, and the same hyperparameter search size, chosen on a held-out real validation split never used for reporting. The downstream segmenter and self-training schedule are frozen across conditions.
Three ablations isolate the key ingredients. The gated translator without gating, which is the same high-resolution architecture with ordinary convolutions and strided streams, isolates leakage prevention. The gated translator with a standard downsampling encoder-decoder isolates resolution. A per-segment affine color map isolates texture translation; it is gating with $1\times1$ kernels, essentially ColorMapGAN made class-aware. A further variant removes the lens-blur model.
Beating the baselines means the following. Boundary-band IoU at $k\le4$ and mean IoU over pole, traffic light, sign, person, rider and bicycle must exceed CycleGAN with semantic consistency and the per-segment color map by more than run-to-run variance over three seeds. Overall mIoU must not drop.
Why not just…?
Why not just match colors per class, ColorMapGAN-style, since that also cannot leak? This is the most serious alternative, which is why its class-aware version is an ablation rather than a strawman. ColorMapGAN's evidence comes from satellite images where source and target share geometry and differ mainly spectrally. MRFP's analysis argues that rendered-to-real gaps include high-frequency texture, which a $1\times1$ map cannot touch. That is argument, not proof. If the per-segment color map matches the gated translator on boundary-band and thin-class IoU, we will conclude that leakage control, not texture translation, carries the benefit, and report the simpler method as the answer.
Why not just add a boundary-preservation loss to CycleGAN using the synthetic masks? A penalty trades off against the adversarial objective, so the generator keeps some leakage whenever leaking fools the discriminator. CyCADA's semantic loss is exactly such a soft constraint, and the colorization literature finds bleeding persists under pixel losses. We run CycleGAN plus the edge-enhancing Sobel loss from the edge-aware colorization work, using synthetic boundaries, as an additional baseline. The leakage score will show directly whether a soft penalty reaches near-zero leakage, and whether residual leakage tracks the boundary-band IoU gap.
Why not skip translation and use feature perturbation or augmentation on a high-resolution segmenter? MRFP shows perturbation alone gives an 8.06-point gain, so this is a real threat. We run MRFP+ on the same HRNet, then our translator, then both combined. If perturbation alone matches translation on boundary bands, translation is unnecessary for this purpose. If the combination wins, the two address different parts of the gap. Our expectation is that perturbation helps broad classes while gated translation helps boundaries, but the experiment decides.
Why not just use HRNet as the segmenter and keep CycleGAN? HRNet preserves resolution in the segmenter. It cannot undo labels that the translator has already misaligned, because the training targets themselves become wrong along smeared silhouettes. The no-gating and downsampling ablations, all paired with the same HRNet segmenter, separate the two effects.
Proposed Experiments
The main setting is unsupervised adaptation from GTA5 to Cityscapes and from SYNTHIA to Cityscapes, reporting mIoU, thin-class IoU and boundary-band IoU. The secondary setting is domain generalization: train on translated GTA5 alone, then test on Cityscapes, BDD100k and Mapillary, matching MRFP's protocol so the 39.27% figure is a meaningful reference.
Four checks are decisive. First, the leakage score must be at the source level for the gated translator and clearly elevated for CycleGAN. Across translators, leakage should predict the boundary-band IoU deficit, which would establish that bleeding in translated training data actually costs accuracy downstream. Second, improvements over CycleGAN must concentrate at small $k$ and on thin classes, shrinking as the band widens. Uniform gains would mean the mechanism is something other than boundary fidelity. Third, the ablations must show whether gating and resolution each contribute. If gating does all the work, the costly high-resolution stream is not justified. Fourth, removing the lens-blur model should change how well the translated images fool a real-versus-synthetic classifier trained on boundary crops. It should also change boundary-band IoU. That would show crisp rendered edges are themselves part of the gap, and that physical blur, unlike semantic leakage, is worth modeling. We also report translator memory and runtime against CycleGAN, since full-resolution gated convolution is the main cost risk.
References
- CyCADA: Cycle-Consistent Adversarial Domain Adaptation
- ColorMapGAN: Unsupervised Domain Adaptation for Semantic Segmentation Using Color Mapping Generative Adversarial Networks
- Multi-Source Domain Adaptation With Collaborative Learning for Semantic Segmentation
- DDColor: Towards Photo-Realistic Image Colorization via Dual Decoders
- Deep Edge-Aware Interactive Colorization against Color-Bleeding Effects
- Playing for Data: Ground Truth from Computer Games
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes
- Deep High-Resolution Representation Learning for Visual Recognition
- MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation