Spread the Errors, Then Cancel Them: Symmetric Losses for Synthetic-to-Real Self-Training

Models trained on synthetic images, such as video-game driving scenes or rendered CAD objects, lose much of their accuracy on real photographs. The most effective fix uses the unlabeled real images directly. The model labels them itself, producing "pseudo-labels," and then trains on its own guesses. Those guesses are often wrong, and ordinary cross-entropy training fits the wrong guesses as faithfully as the right ones. A separate literature offers a principled remedy. A symmetric loss is a loss whose values, summed over all possible labels for a given prediction, always equal the same constant. Under random label noise, the wrong labels then contribute only a constant to the objective, so the noise cannot pull the model away from the clean solution.

The obvious move is to drop a symmetric loss into synthetic-to-real self-training. We argue this will work only partially, and we can say in advance where it will fail. The cancellation argument assumes errors are spread evenly across wrong classes. Synthetic-to-real errors are often systematic: real "riders" get labeled "person," and real "poles" get labeled "building." This note proposes making the pseudo-label errors diffuse on purpose, by sampling labels across strong appearance perturbations instead of taking a consensus. A symmetric loss is then used to cancel the diffused errors. The claim is testable class by class: the benefit of symmetry should track how evenly spread each class's pseudo-label errors are.

Core Claim

The prediction is sharp. On synthetic-to-real benchmarks, swapping cross-entropy for a symmetric or near-symmetric loss on target pseudo-labels should improve only the classes whose confusions are spread out. Classes with one dominant wrong label should gain nothing. Replacing consensus pseudo-labels with labels sampled independently under perturbation should extend the gain to more classes, because it flattens those confusions. Neither ingredient alone should match the combination.

The file contains no prior work that places a symmetric loss inside synthetic-to-real adaptation. Existing adaptation methods take different routes. Consistency training uses cross-entropy (PixMatch). Noisy target labels are filtered with a fitted Gaussian-uniform mixture model (RSDA). Adaptation to unknown classes uses geometric subspace thresholds plus cross-entropy (LEAD). The robust-loss papers test only on noise injected into source labels. These include Symmetric Cross Entropy, asymmetric losses and Joint Asymmetric Loss. The new element here is not the drop-in substitution. It is the diagnosis of when that substitution fails, together with a pseudo-labeling procedure designed to put the noise into the form the loss can cancel.

Why Symmetry Needs Diffuse Errors

Take one target pixel whose true class is $y$, among $K$ classes. Suppose its pseudo-label is wrong with total probability $\eta$. Write the probability of wrong label $j$ as $\eta/(K-1)+\delta_j$, where the deviations $\delta_j$ sum to zero and measure how uneven the confusion is. For a symmetric loss $L$ with $\sum_k L(k,f)=C$, the expected loss on the noisy label is

$$\mathbb{E}\,L(\tilde y,f) = \Big(1-\tfrac{K\eta}{K-1}\Big)L(y,f) + \tfrac{\eta C}{K-1} + \sum_{j\neq y}\delta_j L(j,f).$$

The first term is a positively scaled clean loss whenever $\eta<(K-1)/K$. The second term is a constant. The third term is the only thing that can bias training, and its size is at most $\|\delta\|_1 \max L$.

This makes the failure mode concrete. The classical guarantees (Ghosh et al.; Charoenphakdee et al.) hold when $\delta=0$. Systematic confusion under domain shift is exactly large $\|\delta\|_1$. It is also the regime where the clean-labels-domination assumption behind asymmetric losses breaks. The quantity to drive down is therefore the confusion unevenness, not the noise rate.

The derivation treats labels as fixed. Self-training labels change as the model changes, so we treat the decomposition as a diagnostic to be checked empirically, not as a guarantee.

Research Program

The system builds on PixMatch-style consistency training. It keeps cross-entropy on the clean synthetic source labels and changes only the target branch, in two ways.

The first change is the loss on target pseudo-labels. Pure mean absolute error is known to underfit badly (Memorization in DNNs). We therefore use trainable near-symmetric combinations:

  • Symmetric Cross Entropy, which adds a reverse cross-entropy term to cross-entropy.
  • A normalized "active" loss paired with a "passive" loss, from the Active Negative Loss line.
  • JAL's asymmetric mean-square passive term (Joint Asymmetric Loss).

The second change is how pseudo-labels are produced. For each target image we apply several strong appearance perturbations. These are Fourier low-frequency swaps and CutMix, as in PixMatch, and painting-based global and local texture randomization (GTR/LTR). We take the argmax prediction under each perturbation. The training label for each pixel is drawn at random from these per-perturbation argmaxes, not by majority vote.

Sampling, rather than voting, is the point of the design. Voting re-concentrates the errors onto the most common wrong class. Sampling keeps a disagreeing pixel's label noisy but spreads its wrong mass over whichever classes the perturbations disagree on. The labels stay hard, which is consistent with PixMatch's finding that hard labels beat soft ones. Because labels come from argmaxes rather than confidence scores, the method also avoids the probability-blindness problem: symmetric losses give poorly calibrated scores (Charoenphakdee et al.), and nothing here thresholds on confidence.

Alongside the method, the analysis measures a per-class unevenness score, $\|\delta\|_1$ of the pseudo-label confusion. It is computed on the labeled Cityscapes validation split, for analysis only and never for training or model selection. We also test a label-free proxy for it, based on how many distinct wrong classes the perturbations produce.

The hard-versus-conjugate pseudo-label analysis proves that an even-symmetric objective converges exponentially where non-symmetric losses crawl. That result covers only binary Gaussian mixtures. We use it as motivation that symmetry changes self-training dynamics, not as a guarantee for deep multiclass segmentation.

Baselines

The naive reference point is the source-only model trained on synthetic data. VisDA reports 21.4 mIoU for this on its GTA5-to-Cityscapes segmentation track (VisDA). It fixes how large the gap is.

The strongest directly comparable adaptation method is PixMatch with cross-entropy on hard pseudo-labels and Fourier plus CutMix perturbations. It reports 50.3 mIoU on GTA5-to-Cityscapes and 54.5 mIoU on SYNTHIA-to-Cityscapes (13 classes) (PixMatch). We rerun it from the same code base, with the same architecture, schedule, perturbations and number of training steps as our method. Our reproduction, not only the published figure, is the reference.

A confidence-thresholded variant of the same system, which masks pixels below a softmax threshold, is the second strong competitor. It gets the same tuning budget as our method.

For the domain-generalization comparison, texture randomization without target data reports 43.7 mIoU on GTA5-to-Cityscapes (GTR/LTR).

On VisDA-C classification, the reported numbers are 51.62% for DAN and 92.8% for the challenge winner (VisDA). For RSDA, the spherical robust pseudo-label method, the file lists VisDA among its benchmarks but gives no VisDA figure: no reported result found on Lacuna for RSDA on VisDA-2017. We therefore rerun it under our protocol, or report its Office-31 average of 91.1% only in the Office-31 comparison.

The key ablations form a two-by-two grid: cross-entropy versus symmetric loss, crossed with consensus versus sampled pseudo-labels. This isolates each ingredient.

All variants share hyperparameters selected once, under an identical search budget, on a held-out slice of GTA5 source data. None may use target labels. Beating PixMatch means a mean gain over three seeds that exceeds the seed standard deviation on both segmentation benchmarks, with the gain concentrated in the hard classes the theory predicts.

Why not just…?

Why not just drop Symmetric Cross Entropy into PixMatch's target loss? This is the obvious idea and a real threat, so it is one cell of the ablation grid, not something we argue around. The decomposition predicts it will help diffusely confused classes and stall on classes with a dominant wrong label. If the drop-in matches the full method across classes, the sampling component adds nothing, and we will report that the simpler recipe is sufficient.

Why not just keep cross-entropy and filter by confidence? Thresholds tend to exclude exactly the hard classes that produce low-confidence predictions. Ma et al. describe this class-biased learning under cross-entropy (Symmetric Cross Entropy). LEAD notes that hand-set thresholds are brittle under domain shift (LEAD). The per-class results will show whether thresholding recovers rider, pole and sign IoU. If it does, our advantage shrinks to the easy classes, and we will say so.

Why not just randomize textures more aggressively and keep cross-entropy? Sampled labels with cross-entropy is precisely the cell that tests this. Diffuse noise that cross-entropy can still memorize should leave it behind the symmetric version. If it does not fall behind, flattening the confusion is the whole story and the loss is irrelevant. Stronger randomization also risks destroying semantic cues (GTR/LTR), which we track through clean-source accuracy.

Why not just estimate a noise transition matrix and correct the loss? Pseudo-label noise here depends on the individual image and changes as the model trains. Transition matrices are unidentifiable without anchor points even in the static case (Charoenphakdee et al.).

Proposed Experiments and Decisive Checks

The main experiments are GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes segmentation, scored by mIoU and per-class IoU over three seeds. VisDA-C classification (synthetic CAD to real COCO) serves as a secondary check that the effect is not specific to segmentation.

Three checks decide the claim.

  1. Controlled noise. We train on Cityscapes using its true labels, corrupted with a fixed noise rate $\eta$ but varying unevenness $\|\delta\|_1$. This tests whether the symmetric-loss advantage over cross-entropy shrinks as unevenness grows in a deep segmentation network, as the decomposition predicts.
  2. Per-class correlation in real adaptation. Across the 19 Cityscapes classes, the per-class gain of symmetric loss over cross-entropy should correlate negatively with measured confusion unevenness. Sampled pseudo-labels should both lower unevenness and raise that class's gain. Cross-entropy with sampled labels should not gain comparably.
  3. Label-free proxy. If the unevenness proxy computed without target labels correlates with the true unevenness, the method can in principle switch losses per class in deployment.

If the first two predictions fail, the diffuse-then-cancel account is wrong, whatever the headline mIoU.