Translated Synthetic Labels as Projected Labels

When engineers train a street-scene segmenter on video-game frames, they often first restyle the frames to look like real camera footage. A segmenter labels every pixel with a class such as road, car or pole. The restyling is done with an unpaired image translator, a generative network trained on two unmatched piles of images, one synthetic and one real. The original game label map is then reused, unchanged, as the answer key for the restyled frame. That reuse quietly assumes the translator never moves an edge. In fact translators thicken foliage, erode thin poles, soften car silhouettes and sometimes hallucinate objects. Natural language processing has faced the same situation for years under the name annotation projection. There, labeled English text is machine-translated into another language, and its labels are carried across with a word aligner. The carried labels are wrong in structured, predictable ways, and the useful methods model that structure rather than ignore it.

This note proposes to treat sim-to-real image translation as label projection. The corruption it introduces would be modeled with a learned spatial noise kernel: a small, class-pair-specific distribution over how far and in which direction each class's boundary drifts during translation. The idea extends the cross-lingual noise transition matrix of Learning when to trust distant supervision from "which tag turns into which tag" to "which class bleeds into which neighbor, and by how much." The practical stake is a question the field has left open. Modern transformer self-training (DAFormer) beats every pixel-translation pipeline without translating a single pixel. We want to know whether translation is simply obsolete, or whether its benefit has been hidden by projection noise in the reused labels.

Core Claim

The first claim is that the error between a synthetic label map and the true labeling of its translated image is structured. Specifically, it is concentrated at boundaries, asymmetric between class pairs (for example, vegetation expanding into sky more than sky into vegetation), and specific to the translator used. Because of this structure, a forward-corrected loss with a low-parameter spatial kernel should recover the appearance benefit of translation that raw label reuse wastes.

If that holds, translation plus kernel correction should add measurable mIoU (mean intersection-over-union, the standard segmentation score) on top of a DAFormer-style self-training pipeline, where naive translation adds little or nothing. The claim fails in two ways. It fails if the measured drift is isotropic and class-agnostic. It also fails if a fixed "ignore a band of pixels around every edge" rule matches the kernel. In either case we would report that projection noise is real but trivially handled.

Let $p_\theta(u)$ be the segmenter's clean class distribution at pixel $u$, and $\tilde y$ the synthetic label reused on the translated image. The model explains the reused label through

$$\tilde p_{\tilde c}(u) = \sum_{c} \sum_{\delta \in W} K_{c \to \tilde c}(\delta)\, p_{\theta,c}(u+\delta),$$

where $W$ is a small window of offsets $\delta$ (for example 9×9) and $K_{c \to \tilde c}(\delta)$ is the probability that a pixel truly of class $c$ at offset $\delta$ is recorded as $\tilde c$. The kernel is normalized so that the probabilities over all $\tilde c$ and $\delta$ sum to one for each true class $c$.

Training minimizes cross-entropy between $\tilde p$ and $\tilde y$ on translated frames. At test time, the clean $p_\theta$ is used directly. If $K$ is supported only at $\delta=0$, the model reduces exactly to the spatially invariant class-confusion matrix used by Fang and Cohn and by Label-Noise Robust Domain Adaptation. Both of those papers name instance- and boundary-dependent noise as the case they cannot handle, and that is the case this kernel targets.

Relation to Prior Work

Three sim-to-real translation methods motivate the problem by assuming it away. CyCADA translates GTA5 frames with a cycle-consistent generator. It adds a semantic-consistency loss so the translator does not flip labels, then trains on the original labels as exact ground truth. SPLAT conditions a lighter translator on synthetic masks and reuses synthetic boxes, assuming objects do not move. The CUT robotics study uses a patch-contrastive translator and treats the source masks as fully valid after translation. All three try to keep the translator honest. None of them models the residual dishonesty in the label used for training, which is the step we add.

Two self-training lines constrain the design. DAFormer shows that a hierarchical transformer with rare-class sampling and an ImageNet feature-distance loss reaches 68.3 mIoU on GTA5→Cityscapes with no translation. Any translation proposal must therefore be evaluated on top of this pipeline, not against weaker CNN baselines such as Brehm et al. at 50.1 mIoU. Pseudo-label self-refinement (PRN) learns a per-pixel error mask and refined labels for target pseudo-labels. Its style transfer is low-frequency Fourier amplitude swapping, which does not move edges. PRN is the closest competitor in spirit because it learns where labels are wrong. It differs from our proposal in three ways: it treats error as unstructured per-pixel unreliability, it applies to target pseudo-labels, and it never models directional displacement.

The pseudo-label literature also shapes how the correction is applied. Pseudo-Label Noise Suppression Techniques finds that hard confidence filtering throws away correct low-confidence labels. That finding is why we use forward correction, which keeps every pixel and explains it, rather than masking.

The cross-lingual literature supplies both the diagnosis and the method. FILTER and runtime-MT reading comprehension document projected labels landing at shifted positions. Projection-based temporal labelling shows that aligners err systematically, consistently putting tags on function words rather than content words. That is the kind of structure a learned kernel can capture. Fang and Cohn supply the forward-correction recipe. They also supply a warning: their matrix needed about 1,000 gold tokens to be identifiable. We take that warning seriously and test both an anchored and an anchor-free variant.

Research Program

The first component is measurement, and it does not depend on the method working. We will train CycleGAN and CUT translators from GTA5 to Cityscapes. For roughly 100 translated GTA5 frames, annotators will hand-label the translated images under Cityscapes guidelines. This "projection audit set" lets us measure, for the first time, the full joint distribution of class and displacement between the reused label and the true label of a translated image. From it we will report three things: how much of the disagreement lies within a few pixels of a boundary, how asymmetric the class pairs are, how far it exceeds a uniform erosion model, and how it differs between the two translators. These statistics would be useful to anyone reusing translated synthetic labels, whatever happens downstream.

The second component is the corrected training loss. The kernel $K$ is parameterized per ordered class pair over a 9×9 window. Pairs that never co-occur at boundaries are tied to the identity. We will estimate $K$ in two regimes. In the anchored regime, $K$ is fit on half of the audit set, the vision analogue of Fang and Cohn's gold tokens. In the anchor-free regime, $K$ is learned jointly with the segmenter on translated frames. To keep it identifiable without gold labels, we will add a mass penalty on off-center offsets and initialize it at the identity. Comparing the two regimes tells us whether this kind of supervision requires gold labels in vision, as it did in POS tagging.

The corrected translated-source branch is then plugged into the DAFormer codebase (lhoyer/DAFormer) alongside its unchanged target self-training. This placement tests whether translation adds information that the transformer's shape bias does not already provide.

The third component is a deliberately narrow analysis of error cascade. Kernel-corrected translated labels should yield cleaner target pseudo-labels in the first self-training rounds. Prior work flags those early rounds as the point where errors become locked in (Learning Transferrable Representations for UDA). We will track boundary-region pseudo-label error on Cityscapes validation across training to see whether the gain appears early and persists or fades.

Baselines

The naive reference point is DAFormer itself, run from the official code with no translation. It has a reported 68.3 mIoU on GTA5→Cityscapes (DAFormer). We will reproduce it under our seeds and compute budget, and compare every variant against our reproduction rather than the published figure.

The strongest existing method is PRN, which reports 71.3 mIoU on GTA5→Cityscapes on top of DAFormer and 75.0 with high-resolution HRDA training (Pseudo-label self-refinement). It is the strongest competitor because it already learns where labels are wrong and pairs this with a style transfer that cannot move edges. We will run it in the same DAFormer-based setting, without HRDA unless all methods use it. Our method is also orthogonal enough to combine with PRN: the kernel acts on translated source labels, while PRN acts on target pseudo-labels. Beating PRN therefore means one of two things. Either our variant exceeds PRN's mIoU when run alone, or adding our kernel on top of PRN improves it beyond seed variance (three seeds, with gains reported alongside standard deviation).

Two translation baselines isolate the role of translation itself. "Translated plus raw labels" is CyCADA-style reuse inside DAFormer, run with both CycleGAN and CUT. Fourier amplitude swap plus raw labels is a translation that should carry almost no projection noise, so it shows how much of the translation benefit survives when edges never move.

Three ablations isolate the key ingredient, which is spatial structure. The first is a spatially invariant class-confusion matrix, our kernel restricted to $\delta=0$, as in Fang and Cohn and DCIC. The second is a uniform boundary ignore band, tuned over widths of 1 to 7 pixels on validation. The third is per-pixel dynamic confidence weighting on translated labels, following the noise-suppression paper.

Every method receives the same GTA5 and Cityscapes data, the same translator checkpoints where applicable, the same number of training iterations, and the same validation-tuning budget of eight configurations each. Ours counts as beating a baseline only if it wins under these matched conditions.

Why not just…?

Why not just use DAFormer and skip translation entirely? This is the most serious threat. The 68.3 versus 50.1 mIoU gap between DAFormer and Brehm et al. suggests translation adds little once the backbone is a transformer. Against that, the CUT robotics study moved YCB-Video segmentation from 0.701 to 0.763 mIoU, with real-labeled training at 0.770, so translation does carry signal in some regimes. The "translated plus raw labels" baseline inside DAFormer settles the question. If raw translation adds nothing and kernel-corrected translation also adds nothing, we conclude that projection noise was not the bottleneck and report it.

Why not just use Fourier style swapping, which never moves edges? It avoids projection noise by construction. However, it changes only global color and tone, not texture or sensor-noise statistics, which is plausibly why PRN still needs a separate refinement network. If Fourier swapping with raw labels matches kernel-corrected neural translation, the proposal's premise loses most of its value. We run exactly this comparison.

Why not just ignore a band of pixels around every edge? This simple alternative is a direct competitor to our method's specific contribution. If the audit set shows isotropic, class-agnostic drift, a tuned band should match the kernel. Our method only earns its parameters if drift is asymmetric across class pairs. One example would be pole and sign pixels lost to background while sidewalk and road blur symmetrically. A uniform band handles such asymmetry badly: it either discards too much of thin classes or keeps too much error for wide classes. The audit statistics predict the answer before any training, and per-class IoU on thin classes against the tuned band confirms it.

Why not just run PRN's learned error mask on translated frames? A mask can only say "do not trust this pixel." The kernel says "this pixel is probably the neighboring class," which keeps gradient signal at exactly the boundary pixels a mask throws away. Consistent with this, the noise-suppression paper found that discarding low-confidence pixels hurts. We include the dynamic-weighting baseline and the PRN combination so that this difference is measured, not asserted.

Proposed Experiments and Decisive Checks

The main setting is GTA5→Cityscapes, scored by 19-class mIoU, with per-class IoU and a boundary-band mIoU computed within 5 pixels of ground-truth edges. SYNTHIA→Cityscapes is the replication setting. YCB-Video with CUT is the second domain. Its reported translated-versus-real gap (0.763 vs 0.770) is small, which lets us ask whether kernel correction closes nearly all of the remaining distance. If it does not, that distance is not driven by label projection.

The first decisive check is purely diagnostic. Before any training, the audit set must show that a parametric displacement kernel explains translated-label error substantially better than both a class-confusion matrix and a uniform erosion model, measured by held-out likelihood on the audit set's second half. If it does not, we stop the method and publish the measurement.

The second check asks whether translation is useful at all once its labels are corrected. Kernel-corrected translation inside DAFormer must beat both our DAFormer reproduction and Fourier swapping with raw labels on mIoU, and the gain must concentrate in boundary-band and thin-class IoU.

The third check is anchor dependence. If only the anchored regime works, the method inherits Fang and Cohn's cost of hand-labeling around 100 images. That is still cheap for vision, but it changes the claim from unsupervised to minimally supervised.

The fourth check is translator dependence. Across CycleGAN and CUT checkpoints taken at several training stages, we will plot learned kernel off-center mass against gain. If the method works as claimed, gains should grow with how much the translator moves edges. If the gains do not follow kernel mass, the improvement comes from something other than modeling projection noise.