When synthetic street scenes from a video game are restyled to look like real photographs, the restyled images keep their original pixel labels. That only helps a segmentation model if the restyling does not quietly change what is in the picture. Often it does: sky turns into foliage, sidewalks turn into road, thin poles disappear, and the labels become wrong. Adaptive instance normalization (AdaIN) is a cheap, layout-preserving way to restyle. It rescales each feature channel so its mean and standard deviation match those of a target image. Its weakness is that it uses one set of statistics for the whole image, so rare classes inherit the appearance of dominant ones. The natural fix is to match statistics class by class. On real target images, however, nobody knows which pixels belong to which class, and wrong guesses are exactly what corrupt the statistics.
We propose to treat that unknown class assignment the way satellite positioning treats its unknown whole-number cycle counts. A receiver first estimates the count as a real number. It then removes known hardware offsets, which would otherwise shift the estimate away from the true integer. Only after a validation test passes does it commit to an integer, and it leaves the remaining unknowns unresolved. Carried over to restyling, this gives three steps. We remove a per-image appearance offset before deciding classes. We accept a class decision for a target region only when a margin test, calibrated to a chosen failure rate, says the decision is reliable. Only accepted regions contribute to a bank of class-wise target statistics, and that bank drives class-wise AdaIN on source images. The same test then audits each translated source image and flags pixels whose label the translation appears to have flipped.
Core Claim
Class-conditional AdaIN fails or succeeds depending on how reliable the target class statistics are, not on the normalization operator itself. Gating those statistics with bias-removed, failure-rate-calibrated commitment should cut label flipping on GTA5/SYNTHIA→Cityscapes. It should help most on thin and rare classes, and it should beat both global AdaIN and class-wise AdaIN built from raw pseudo-labels. Here "pseudo-labels" means the target-image class predictions of a segmentation model trained only on source data. The positioning literature contributes a recipe for when to trust a discrete decision. It does not contribute a lattice-search algorithm. We do not claim that integer least-squares search is needed for image labels.
Research Program
Prior work supplies the motivation, the method pieces and the constraints. SRUNIT and its extension, Lipschitz Regularized CycleGAN, show that distribution matching flips labels when class frequencies differ between domains. Their fix is a smoothness penalty on the generator. Ours removes the cause instead: we stop matching statistics across classes at all. DCAN supplies the operator: it applies global AdaIN at both the image and feature level and reaches 41.7% mIoU with PSPNet on GTA5→Cityscapes. We keep its encoder–decoder but make the statistics class-specific. RAIN shows that region-restricted statistics help harmonization. It needs ground-truth masks and handles only foreground versus background. We need many classes, and we must work without target masks. RAIN also constrains our design, because it found that modulation must sit in the decoder to avoid damaging semantic features. We follow that placement.
The first component is per-image bias removal by differencing. In double-differenced positioning, subtracting paired observations cancels offsets shared by those observations (PPP–RTK review). By analogy, we describe each target class by its offset from the image-wide statistic rather than by its absolute statistic:
$$\delta_k(y) = \mu_k(y) - \mu(y), \qquad \rho_k(y) = \log \sigma_k(y) - \log \sigma(y),$$
Here $\mu_k(y)$ and $\sigma_k(y)$ are the channel-wise mean and standard deviation of decoder features over the pixels of target image $y$ assigned to class $k$, and $\mu(y)$, $\sigma(y)$ are the same quantities over the whole image. A global exposure or white-balance shift moves every class together, so it largely cancels in $\delta_k$ and $\rho_k$. Class decisions on target images are also made after this normalization: features are instance-normalized before the source-trained segmentation model scores them.
The second component is commitment with a validation test. The source-trained model scores each target superpixel, and we accept the top class only if the ratio between the top two class scores exceeds a threshold. That threshold is set to hit a target failure rate (say 2%), echoing the controllable-success-rate validation used before fixing ambiguities. Real target labels are unavailable, so we calibrate on already-translated source images, where labels are known. Whether this proxy transfers is an empirical question we test directly.
The third component is shrinkage for small regions. A class can be accepted in only a few pixels of an image, and its variance estimate is then unstable. This is the analogue of the ill-conditioned short-window systems that the substation robot study stabilizes with Tikhonov regularization. We shrink each class statistic toward the global one with weight $n_k/(n_k+\lambda)$, where $n_k$ is the class's accepted pixel count.
The final component is the translation and audit step. For a source image, we normalize its decoder features within each ground-truth class region. We then re-style them using a reference target image's global statistics plus offsets $\delta_k, \rho_k$ drawn from the bank. Classes with no accepted statistics fall back to global AdaIN. A frozen source-trained model, run under the same acceptance test, then checks the translated image. Pixels whose accepted class contradicts the source label are dropped from the downstream loss, and pixels with no accepted class keep their source label. This targets the open problem that labels stay fixed while translated content drifts.
Baselines
The naive reference point is source-only training, together with untranslated source images fed to the same downstream pipeline. CyCADA reports 54.0% source-only per-pixel accuracy on GTA5→Cityscapes (CyCADA).
The closest operator-level baseline is DCAN's global AdaIN, re-run with our decoder, data and training schedule. The reported reference is 41.7% mIoU with PSPNet. Beating it means a clear mIoU gain at matched compute, concentrated on rare classes.
The key ablation is class-wise AdaIN driven by raw, ungated pseudo-labels with absolute statistics. Call it "RAIN with predicted masks." It differs from our method only in the gating, the differencing and the shrinkage. If the full method does not beat it, the positioning-derived ingredients contribute nothing. We also ablate each of the three ingredients separately.
For translation with semantic constraints, we compare against CyCADA, which reports 83.6% per-pixel accuracy versus a 93.1% oracle, and against SRUNIT. For SRUNIT, the file reports results only on Cityscapes label→image (27.23 mIoU versus 23.43 for CUT). No reported GTA5→Cityscapes result was found on Lacuna for SRUNIT, so we will run it ourselves.
The strongest existing system is HRDA, at 73.8 mIoU on GTA5→Cityscapes and 65.8 on Synthia→Cityscapes. On top of it, ProCST used as preprocessing reaches 73.9 and 66.4. Our translated images will replace ProCST's as HRDA input, using the DAFormer codebase with identical seeds, iterations and tuning budget. Beating ProCST+HRDA means exceeding 73.9 and 66.4 over at least three seeds, at a fraction of ProCST's progressive multi-scale training cost.
Why not just…?
Why not just threshold pseudo-label confidence? A ratio test on the top two scores is close to a margin threshold, and we will not pretend otherwise. What differs is that we test after removing the per-image bias, and that we set the threshold by a calibrated failure rate rather than by hand. The ablation compares gated-but-not-differenced against full; if a hand-tuned confidence threshold matches the full method, we will report the contribution as shrinkage plus differencing only.
Why not just add a semantic consistency loss to global AdaIN? CyCADA's loss penalizes flips softly. SRUNIT shows that soft penalties still trade semantics for distribution matching when class frequencies differ, and CyCADA itself needed staged training for memory reasons. Global AdaIN plus a consistency loss is run as an additional baseline. The decisive comparison is the flip rate described below.
Why not just skip translation and run HRDA? This is the real threat. ProCST's gain over HRDA was only about one point, so input-level translation may matter little to a strong self-training pipeline. We therefore report two numbers separately. The first is the translated-only setting, where translation quality dominates. The second is the HRDA setting, where we expect at most modest gains. If the HRDA gain is nil, the contribution is a cheaper and more faithful translator with a label audit, not a better final mIoU.
Proposed Experiments
We run GTA5→Cityscapes and SYNTHIA→Cityscapes as the main settings, and the VisDA 2022 industrial-waste split (where ProCST reports +2.2 mIoU) to test transfer to non-urban scenes.
The decisive check is a direct measure of label flipping. A segmentation model trained on real Cityscapes, used only for evaluation and never for training, labels each translated GTA5 image. We count disagreement with the source label per class, focusing on pole, traffic sign and sky. Our method should cut the flip rate relative to global AdaIN and to ungated class-wise AdaIN, and this should come with per-class IoU gains on those classes.
Two further checks probe the borrowed ingredients. The first tests whether the calibrated 2% failure rate on the source proxy matches the realized error of accepted target regions, as measured with real Cityscapes labels on a held-out split. The second tests whether the differenced statistics vary less across target images than the absolute ones, which is the premise borrowed from differencing.
The audit is evaluated by asking what fraction of the pixels it drops are true flips according to the evaluation model, and whether dropping them improves downstream mIoU. Downstream segmentation reports mIoU in both the translated-only and the HRDA-preprocessing settings, along with translation wall-clock cost.
References
- Semantically Robust Unpaired Image Translation for Data with Unmatched Semantics Statistics
- DCAN: Dual Channel-Wise Alignment Networks for Unsupervised Scene Adaptation
- Region-Aware Adaptive Instance Normalization for Image Harmonization
- Review of PPP–RTK: achievements, challenges, and opportunities
- Path planning of substation inspection robot based on high-precision positioning and navigation technology
- CyCADA: Cycle-Consistent Adversarial Domain Adaptation
- HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation
- A simple preprocessing approach for improving semantic segmentation in unsupervised domain adaptation