When a simulator renders a street scene, it produces a perfect label for every pixel. That label is the main reason to use synthetic data at all. To make the synthetic image look real, practitioners usually pass it through an unpaired image translator, meaning a generator trained to make synthetic images resemble real photographs without ever seeing matched pairs. These translators often quietly change what the scene contains. They turn sky into buildings, erase traffic signs, or bend organ boundaries, and the original label then describes an image that no longer exists. Our proposal starts from a simple observation. Restyling a scene takes little information per pixel: color, lighting, sensor noise, and material texture are low in detail or spatially smooth. Changing what an object is requires the generator to inject a lot of new, scene-specific structure at that location. If we give the generator a measured, per-pixel budget of new information, and let a learned spatial mask decide where that budget is spent, semantic changes become expensive and visible, while realistic restyling stays cheap.
The core claim is that the injected information rate, measured per pixel, can do two jobs. It is a control knob that trades realism against label fidelity. It is also a map of where the synthetic label can no longer be trusted, and the downstream segmentation network can use that map directly by down-weighting pixels where the translator spent many bits. Existing fixes act globally, by rebalancing class statistics or adding a consistency loss. This approach adds a local, measurable account of where drift happened and uses it during task training.
What the prior work establishes
Several papers motivate the gap. Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics shows why drift happens. When the target domain has more of some class than the source, an unconstrained generator relabels under-represented source regions to match the target's class frequencies. Its remedy enforces robustness of the translation to small perturbations. Our remedy instead limits how much new content can enter each location. KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes documents hallucination and deletion of obstacles and signs in driving scenes. It attacks the cause by estimating density ratios between domains to correct the statistical mismatch. That is a global correction applied to training data or losses. It says nothing about which specific pixels of a given translated image were corrupted, and that per-pixel output is what we add.
Two further papers show the problem outside static driving scenes. Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation reports large regions being overwritten by dominant target classes, such as sky becoming buildings. Exploring semantic consistency in unpaired image translation to generate data for surgical applications shows geometric deformation of anatomical margins in a surgical setting. We treat these two papers as evidence that drift takes several forms: whole-region relabeling, small-object deletion, and boundary deformation. Any fix must handle all three, and global statistic matching mainly addresses the first.
The research file available to us was truncated before its survey of closest existing work. We therefore cannot rule out prior spatially masked or bottlenecked translators beyond those cited here. The distinguishing element we commit to is the use of a measured per-pixel rate as a downstream label-trust signal. Masking alone, or a global bottleneck alone, does not provide this.
Core mechanism
The generator is split into two paths. The first, $A(x)$, is a deliberately weak appearance path. It has a small receptive field, so it can change color, contrast, and fine texture of the synthetic image $x$ but cannot redraw object-scale shapes. The second path decodes a stochastic spatial latent $z$ through a decoder $D$. Each latent cell $z_i$ is sampled from an encoder distribution $q_\phi(z_i \mid x)$, and its KL divergence from a fixed prior $p(z_i)$ upper-bounds the information it carries. A spatial attention mask $m \in [0,1]$, predicted from $x$, gates where the second path may write:
$$ y = A(x) + m \odot D(z), \qquad r_i = \mathrm{KL}\big(q_\phi(z_i \mid x)\,\|\,p(z_i)\big). $$
Here $y$ is the translated image and $r_i$ is the rate, in nats, spent at location $i$. Training minimizes an adversarial realism loss plus a rate penalty:
$$ \mathcal{L} = \mathcal{L}_{\text{adv}}(y) + \lambda \sum_i \beta_{c(i)}\, r_i , $$
where $\lambda$ sets the overall budget and $\beta_{c(i)}$ is a per-class price looked up from the synthetic label $c(i)$ at pixel $i$. Using the synthetic label here is legitimate because the source labels are exact. Safety-critical small classes such as signs, pedestrians, and poles get a high price, which makes deleting them expensive. The mask lets the generator concentrate its budget where the simulator is visibly unrealistic, for example on vegetation texture, without paying for regions that need only restyling.
The rate map then feeds downstream training. The segmentation network trained on translated images weights its per-pixel cross-entropy by $w_i = \exp(-r_i/\tau)$, where $\tau$ is a temperature. Pixels where the translator injected many bits, and therefore may have changed their meaning, contribute less supervision.
Research Program
The program has three parts. First, we build the translator and verify that the weak appearance path alone cannot produce object-scale relabeling, by measuring drift with the stochastic path disabled. Second, we test the central empirical hypothesis: that the per-pixel rate $r_i$ predicts where drift occurs. We measure drift at the pixel level by running a segmentation network trained only on real target images over the translated images and comparing its output to the original synthetic labels. We then report how well $r_i$ ranks drifted against faithful pixels, measured by area under the ROC curve. Third, we sweep $\lambda$ to trace a realism-versus-fidelity curve, and we test whether rate-based trust weighting improves the downstream model beyond what the budget achieves alone.
Baselines
The naive reference point is source-only training: a segmentation network trained directly on untranslated synthetic images and evaluated on real validation data. Every translation method must beat it to justify its cost. The second reference is an unconstrained unpaired translator with the same generator capacity and no rate penalty or mask. It is the direct producer of drift and defines how much label corruption we are removing.
The strongest existing competitors in the file are the perturbation-robust translator of Semantically Robust Unpaired Image Translation and the density-ratio method of KLIEP-based Density Ratio Estimation. Both are strongest on this question because they target exactly the class-statistics mismatch that causes drift. We will rerun both from their described procedures on the same synthetic and real splits. Each gets the same generator backbone where the method allows, the same segmentation network and schedule, and an equal hyperparameter search budget, which we fix at the same number of trials per method selected on a held-out real validation split. The file gives no numbers, so we report: no reported result found on Lacuna for either method's downstream segmentation score on the driving benchmark. We will establish their values ourselves. Beating them means higher real-domain mean intersection-over-union, together with a lower small-object deletion rate, at matched or better realism score. A gain in mean IoU that comes with more deleted signs would not count.
Ablations isolate each ingredient. The first removes the mask, so the budget is uniform in space. The second sets $\beta$ uniform across classes. The third removes trust weighting while keeping the budgeted translator, and the fourth keeps trust weighting but computes it from mask values rather than rates. The third and fourth ablations are decisive for the central claim that rate is the right trust signal.
Why not just…?
Why not just add a semantic consistency loss using a segmenter trained on synthetic data? This is the most common fix. It asks a synthetic-trained segmenter to still read the source labels on the translated image. That segmenter is unreliable on real-looking inputs, which is the very domain gap in question, so it can penalize correct restyling and miss plausible-looking relabels. We include it as a baseline under the same tuning budget. If it matches our method on both mean IoU and deletion rate, the budget adds nothing and we will say so.
Why not just rebalance class statistics, as the density-ratio approach does? This is a real threat for whole-region relabeling such as sky becoming buildings, because it removes the pressure that causes that drift. Rebalancing is global, however, so it offers no protection against deleting a single rare sign in one image and produces no per-pixel trust map. The experiments settle this by breaking results down by drift type. If rebalancing equals our method on small-object deletion too, the proposal's value shrinks to the trust map, and we test that separately by applying trust weighting on top of the rebalanced translator.
Why not just use a low-capacity translator such as color or spectrum transfer, and skip generative translation? Our appearance path $A(x)$ alone is essentially this baseline. It cannot drift at object scale, but it may leave a large realism gap on textures. Reporting $A$-only results tells us whether the stochastic budgeted path earns its complexity.
Why not just fix the mask from labels, translating only chosen classes? A hand-set mask needs a priori knowledge of which classes look unrealistic, and it still gives unlimited capacity inside the masked regions. The uniform-$\beta$ and learned-versus-fixed-mask ablations show whether learned allocation matters.
Proposed Experiments
The main setting is synthetic-to-real urban driving segmentation, as studied in the KLIEP and semantic-robustness papers. We will use the standard game-rendered-to-city-photograph pairing and align with those papers' protocols where they are recoverable. A second setting is surgical synthetic-to-real translation following Exploring semantic consistency in unpaired image translation, where boundary deformation dominates. An optional video setting following Preserving Semantic and Temporal Consistency tests whether per-frame rates stay stable over time.
The metrics are real-validation mean IoU and per-class IoU, pixel drift rate against a real-trained oracle segmenter, small-object deletion recall, a distribution-level realism score of the Fréchet type computed on the translated images, and rate-versus-drift AUROC.
The decisive checks come in three parts. First, rate must predict drift with AUROC clearly above that of mask values and of discriminator confidence. Second, the $\lambda$ sweep must show a curve that dominates the unconstrained translator: fewer deletions at comparable realism. Third, trust weighting must improve mean IoU over the same translator without weighting. If rate does not predict drift, the central claim fails regardless of the downstream numbers.
References
- Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics
- KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes
- Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation
- Exploring semantic consistency in unpaired image translation to generate data for surgical applications