Bounded Pseudo-Label Losses for Synthetic-to-Real Segmentation

Models trained on video-game frames, such as Grand Theft Auto V (GTA5), and adapted to real street scenes learn largely from their own guesses. A "teacher" network labels each pixel of unlabeled real images, and a "student" is trained to match those labels. When the teacher is wrong, standard cross-entropy pushes hardest on exactly the pixels where the student most disagrees with the label, so a wrong label pulls on the weights without limit. This matters most in segmentation, where errors arrive in large contiguous patches (a whole sidewalk labeled "road"). We propose replacing that unbounded penalty with a loss whose pull fades to zero once the student finds a label implausible, and we set the fade-out point separately for each class from the student's own statistics on real images, without any target labels.

The opportunity is concrete. The strongest dense method in our file, masked image consistency (MIC), still trains its student with plain cross-entropy on teacher labels. Its authors note that it struggles where context is uninformative, which is where the teacher makes confident errors. Bounded losses have been tried in domain adaptation only for image classification, and always folded into a larger system. Nobody has measured whether they help, or hurt, when labels are dense, spatially correlated, and wrong in systematic class-specific directions. The study answers that question and also tests the known danger of bounded losses: they ignore signals the model does not already believe.

Core Claim

In student-teacher self-training for synthetic-to-real segmentation, wrong pseudo-labels should concentrate among pixels where the student assigns low probability to the teacher's label. This holds especially under MIC's masking, because the masked student lacks the shortcut cues that misled the teacher. If that is true, a loss that saturates in this low-probability region removes most of the gradient coming from wrong labels while keeping most of the gradient from correct ones.

We use a one-parameter family that runs from cross-entropy to the bounded log-likelihood loss of Don't Force the Fit, which proposed $\log(2-p)$ for language-model fine-tuning:

$$L_a(p) = \log(1+a) - \log(a+p),$$

Here $p$ is the student's probability for the pseudo-label at a pixel and $a>0$ sets where the loss flattens. As $a \to 0$ this recovers cross-entropy, and $a=1$ is the bounded log-likelihood loss. The gradient with respect to the label's logit has magnitude $p(1-p)/(a+p)$, which follows from the softmax derivative $dp/dz = p(1-p)$. It behaves like cross-entropy when $p \gg a$ and fades as $p/a$ when $p \ll a$. So $a$ is roughly "the probability below which a pixel stops counting."

The idea that is new here is how we choose $a$. That paper's own failure mode was knowledge injection: it scored 0 when the model had to learn facts it initially found implausible. In segmentation the analogue is a class whose real-world appearance differs sharply from its rendered appearance, so the student starts with low probability everywhere for that class. A single global $a$ would silence such a class. We therefore set $a_c$ for each class $c$ to a running low quantile of the student's probability on class-$c$ pseudo-labels in recent target batches. This places the plateau below the class's typical agreement level. Hard classes keep learning, while within each class the outlying disagreements are muted. The rule needs no target labels, which addresses the threshold-tuning problem raised in Can gradient clipping mitigate label noise?.

Relation to Prior Work

Three papers motivate the mechanism. Can gradient clipping mitigate label noise? shows that ordinary gradient clipping only makes the loss linear for large errors, which is not enough under label noise, whereas truly saturating losses resist symmetric noise. Its theory explicitly leaves asymmetric noise open. Memorization in Deep Neural Networks shows that symmetric bounded losses stop networks from memorizing random labels where dropout and weight decay do not. Don't Force the Fit supplies the loss form and also its main constraint, the knowledge-injection failure that our per-class plateau is designed to avoid.

Two papers are the closest existing combinations, and we must differ clearly from both. MIXUNBOT treats optimal-transport mismatches as noisy labels and uses symmetric cross-entropy together with MixUp and unbalanced transport, evaluated on VisDA-2017 classification, Office-Home and digits. Its ablation shows the bounded loss alone gave little (about 94.6% versus 94.2% on SVHN→MNIST), and the gain appeared only in combination. RSDA down-weights pseudo-labels using a Gaussian-uniform mixture fitted in a normalized feature space, again only for classification.

Our work differs from both in four ways. It targets dense segmentation. It uses no transport solver and no fitted mixture model. It sets the bound per class from the student's own probabilities. And it begins with a direct measurement of where wrong labels sit, rather than assuming bounded losses help.

The MIXUNBOT ablation also cuts against us, because it suggests a bounded loss by itself may be weak. We address this head-on by building on MIC. MIC already provides a strong regularizer, so it is the setting where MIXUNBOT's pattern of synergy with regularization would predict a gain.

Research Program

We build on the public MIC code (lhoyer/MIC) and change only the student's loss on teacher labels. MIC's existing teacher-confidence weighting stays in place in the main variant.

The first stage is diagnostic, and it can stop the project. During a standard MIC run on GTA5→Cityscapes we use the Cityscapes ground truth, for analysis only, to record three things at each checkpoint: whether each pixel's pseudo-label is correct, the student's probability $p$ for it, and that pixel's share of the total gradient norm. If wrong labels are not clearly enriched at low $p$, the proposal fails, and we report that as a negative result about bounded losses in self-training. The diagnostic also measures how asymmetric the noise is, as a class-by-class confusion of pseudo-labels against truth, which tests the file's claim that the noise is directional.

The second stage compares loss designs at matched compute: plain cross-entropy, global $a$ (including $a=1$), the per-class $a_c$ rule, symmetric cross-entropy, and partially Huberised cross-entropy. The third stage studies knowledge injection. For each class we plot how quickly its target accuracy rises from its source-only starting level, under global versus per-class bounds, to see whether global bounding freezes classes that begin far from correct.

Baselines

The naive reference point is a source-only model trained on GTA5 with no adaptation. The VisDA challenge reports 21.4 mIoU (mean intersection-over-union) for source-only on its GTA5→Cityscapes segmentation track, with an older backbone. We retrain source-only on our backbone so the gap is measured fairly.

The strongest existing method is MIC, which reports 75.9 mIoU on GTA5→Cityscapes (MIC). It is the right competitor because it is the best dense method in the file and is exactly the pipeline we modify. We rerun MIC from its released code with three seeds on the same hardware and schedule. Our variants use the identical schedule, augmentations and masking ratio, and they receive no extra tuning budget, because the per-class rule has no free target-tuned parameter. Beating MIC means a mean mIoU gain larger than the seed-to-seed spread of our MIC reproduction, consistent across all three seeds, with no class losing more than its own seed spread.

A second strong baseline is symmetric cross-entropy dropped into MIC in place of cross-entropy. This is the bounded loss MIXUNBOT chose, so it is the most direct prior-work competitor on the "bounded loss in adaptation" question. It receives the same budget.

Partially Huberised cross-entropy from Can gradient clipping mitigate label noise? is included because it had the best noisy-label results reported in the file: 18.5% for PHuber-CE versus 11.4% for cross-entropy on CIFAR-100 with 60% noise. Its threshold is chosen with the same unsupervised proxy we allow ourselves, never with Cityscapes labels.

The key ablation compares a global $a$ against per-class $a_c$, with $a$ chosen from a sweep. To keep the comparison favorable to the global variant, the sweep is scored on Cityscapes validation labels. If per-class $a_c$ cannot match a label-tuned global $a$, the per-class rule is not contributing anything.

Why not just…?

Why not just raise the confidence threshold on pseudo-labels? This is the most serious threat, because MIC and BDL already filter pseudo-labels by teacher confidence. A stricter filter might remove the same wrong pixels. The two signals differ, though. A threshold uses the teacher's own confidence, and the teacher is miscalibrated under domain shift, as RSDA argues (Spherical Space Domain Adaptation). The bounded loss uses the masked student's disagreement with the teacher. We settle this with a matched-budget test: we tune a teacher-threshold baseline so it discards the same fraction of gradient mass as our loss, then compare both the precision of the discarded pixels (using the diagnostic labels) and the final mIoU. If the threshold matches us, we will report that the bounded loss adds nothing beyond filtering.

Why not just clip gradients? Menon et al. show that clipping is equivalent to a Huberised loss that stays convex. On CIFAR-100 with 60% noise, clipping cut cross-entropy accuracy from 11.4% to 9.0%, and it fell to chance on their synthetic 45%-noise problem (Can gradient clipping mitigate label noise?). Clipping also acts on the whole batch's gradient norm, not on individual pixels, so it cannot separate a wrong sidewalk patch from a correct road patch in the same image. We still run global norm clipping inside MIC as a cheap control.

Why not just use MIXUNBOT's symmetric cross-entropy, or RSDA's mixture weighting? Symmetric cross-entropy is a real competitor and is run directly as a baseline. Its weakness for our setting is that its bound sits at a fixed place for every class, so it should share the knowledge-injection failure on classes that start far from correct. The per-class learning-curve analysis tests this directly. RSDA's mixture assumes features form tight clusters per class on a sphere, and it was never run on dense prediction. Fitting such a mixture over millions of correlated pixels is costly, and we report it only if a pixel-level port proves feasible.

Why not just render more diverse synthetic data? Driving in the Matrix found that 200k GTA5 frames beat real Cityscapes training on KITTI vehicle detection (0.6803 versus 0.6247 mAP on Easy), yet the authors still report a residual domain gap. Large synthetic sources in VisDA still leave source-only segmentation at 21.4 mIoU. Our approach is complementary to more data and does not compete with it.

Proposed Experiments

The main benchmark is GTA5→Cityscapes on the MIC codebase, scored by mIoU over 19 classes, with three seeds for each method. The decisive checks run in order. The first is the stage-one diagnostic: the enrichment of wrong labels at low student probability, and the share of gradient that wrong labels contribute under cross-entropy versus $L_{a_c}$. If our loss does not cut the wrong-label gradient share substantially while keeping most of the correct-label share, we stop. The second is the full comparison against MIC, symmetric cross-entropy, partially Huberised cross-entropy, gradient clipping, and the matched threshold. The third is the per-class analysis. Its prediction is that global bounding hurts the classes with the lowest source-only accuracy, while per-class bounding does not.

To test generality beyond one benchmark, we evaluate on DarkZurich, where MIC reports a +4.3 mIoU gain but the file gives no absolute value. There, nighttime appearance should make teacher errors denser. We also evaluate on VisDA-2017 classification, where MIC reports 92.8% and MIXUNBOT 71.3%. That last test checks whether the effect holds when the spatial clustering of errors is absent. Our expectation is that the gain shrinks for classification. If it does not, our proposed explanation, that clustered dense errors are what bounding exploits, would be wrong even if the method still helped.