Learning What Real Images Are Not: Negative Pseudo-Labels for Synthetic-to-Real Training

Training perception models on rendered data is cheap. A game engine or simulator hands you perfect labels for every pixel. Yet models trained this way break on real images, because they learn simulator-specific cues: clean textures, uniform lighting, rendering artifacts. The standard fix is self-training. A model trained on labeled synthetic images guesses labels for unlabeled real images, called pseudo-labels, and retrains on its own confident guesses. This approach has a blind spot. Under a simulator-to-reality shift, the model's top guess on a real image is exactly what those simulator cues corrupt. So most real images fall below the confidence threshold and get thrown away, often the hard ones: poles, riders, distant cars.

We propose to keep those discarded real images and teach from them, using what the model is sure they are not. A network that cannot decide whether a dim real pixel is a pole or a sign is still almost certainly right that it is not sky or road. Our central claim is that these negative labels stay reliable under synthetic-to-real shift far longer than the top-1 guess does. We further claim that the level of confidence counted as "uncertain" must be judged relative to each domain and class. Applying a single global confidence scale to clean synthetic images and noisy real ones mislabels most real images as hard. We would put both pieces into one end-to-end objective, trained in a single stage, and test them on GTA5/SYNTHIA→Cityscapes segmentation and on synthetic-CAD→real-scan point-cloud classification.

Core Claim and What Is New

The two research directions meet here in a specific way. A unified semi-supervised objective adds a supervised loss on labeled data to an unsupervised loss on unlabeled data and optimizes both at once. The synthetic-to-real gap sets the regime: labeled data is entirely synthetic, and unlabeled data is real.

Existing work at this intersection makes one of two choices. Some methods treat the unlabeled target with a symmetric confidence-sharpening term. Point-Syn2Real (Point-Syn2Real) uses Shannon entropy on real LiDAR scans. Maximum Squares Loss (Domain Adaptation for Semantic Segmentation With Maximum Squares Loss) bounds the entropy gradient so easy classes do not dominate. Other methods rely on staged pipelines with real labels: Dual-Level Domain Mixing (Semi-Supervised Domain Adaptation Based on Dual-Level Domain Mixing) uses two teachers, offline distillation and iterative retraining, and needs 100 or more labeled real images.

None of these methods extracts a separate, asymmetric signal from low-confidence real samples. Complementary labeling for semi-supervised learning (Boosting Semi-Supervised Learning with Contrastive Complementary Labeling) showed that the lowest-probability classes of uncertain samples carry usable information. It reports +2.43% over FixMatch on CIFAR-10 with 40 labels. It was tested only where labeled and unlabeled images come from the same distribution. CalibrateMix (On the Calibration of Image Semi-Supervised Learning Models) scores sample difficulty by prediction margin, also only within a single domain.

Our contribution is to bring the negative-label idea into the synthetic-to-real setting. We argue it needs one change to work there. Margins on real data are systematically depressed by the domain gap. "Uncertain" must therefore mean low-ranked within the real domain and the predicted class, not low on an absolute scale.

The Objective

We use a mean teacher: a slowly updated copy of the network, kept as an exponential moving average of its weights, that produces stable targets. For each real pixel or sample $x$, the teacher gives class probabilities $q(x)$ and a margin $m(x)$, the gap between its top two probabilities. We keep a running bank of real-domain margins for each predicted class $c$. From it we compute the domain-relative margin $r(x)$, which is the quantile of $m(x)$ within that bank. Real samples with $r(x) \ge \rho$ receive an ordinary positive pseudo-label. The remaining real samples receive a negative label set $N(x)$: the $k$ classes the teacher ranks lowest.

The student, with probabilities $p$, minimizes

$$ \mathcal{L} = \mathcal{L}_{\text{CE}}(x_{\text{syn}}, y_{\text{syn}}) + \lambda_{+}\sum_{x:\, r(x)\ge\rho} \mathcal{L}_{\text{CE}}\big(p(\tilde{x}), \arg\max q(x)\big) + \lambda_{-}\sum_{x:\, r(x)<\rho}\ \sum_{c\in N(x)} -\log\big(1-p_c(\tilde{x})\big), $$

where $\tilde{x}$ is a strongly augmented view of the real input. The augmentation includes cross-domain region mixing: a rectangle of a synthetic image with its true labels is pasted into the real image, following the region-level mixing of Chen et al. (Dual-Level Domain Mixing). Here it runs inside one training step rather than across separate teachers.

The negative term pushes a single class probability toward zero. Its gradient stays small once $p_c$ is already small. This keeps it from flooding the update the way entropy does on confident easy classes, the failure mode Chen et al. identified (Maximum Squares Loss). The method needs no real labels. When labeled real images exist, they simply join the first term.

Research Program

The work has three stages, and the first can stop the project.

The first stage is a diagnostic on known benchmarks before any new training. We take a source-only model and a standard self-trained model on GTA5→Cityscapes. On real target pixels, we plot two precisions as a function of the domain-relative margin $r$: that of the top-1 pseudo-label, and that of the $k$-lowest negative labels. We do the same on ModelNet/ShapeNet→ScanNet. The hypothesis is that in the low-$r$ band, negative-label precision stays above roughly 95% while top-1 precision collapses. If negative labels are no more reliable than top-1 guesses under shift, the idea is wrong and we stop. In the same stage we test the margin-shift premise directly. We check whether absolute margins of real samples sit far below those of synthetic samples of the same class, and whether global margins misclassify which real samples are hard.

The second stage trains the objective above in a single stage. It covers the zero-real-label setting (GTA5→Cityscapes and SYNTHIA→Cityscapes, 3D-CAD→ScanNet) and the setting with 100 real labels (GTA5→Cityscapes).

The third stage analyzes failure. Per-class IoU shows whether the gains concentrate on thin and small classes, which is where we expect them. We measure target-domain expected calibration error, the gap between predicted confidence and actual accuracy, to check that negative labels do not produce overconfidence. We also sweep $k$ and $\rho$ to map how sensitive the method is to these settings, a known weakness of semi-supervised objectives.

Baselines

Source-only. The do-nothing reference is the same segmentation network trained only on synthetic labels. Point-Syn2Real reports the analogous 3D figure of 19.76% on 3D_City→SemanticKITTI (Point-Syn2Real). We rerun the 2D version ourselves.

Published target objectives. The symmetric-objective baselines are entropy minimization in the Point-Syn2Real style and Maximum Squares Loss. The latter reports 46.4% mIoU on GTA5→Cityscapes (versus ADVENT at 44.8% and CBST at 46.2%) and 48.2% on SYNTHIA→Cityscapes over 13 classes (Maximum Squares Loss). These are the strongest published unsupervised numbers in our file. We reimplement both on our backbone rather than quote them, because a modern backbone will move all numbers.

Strongest practical baseline. The competitor that matters most is our own pipeline with the negative term removed. It keeps the mean teacher, the fixed confidence threshold and the cross-domain region mixing. This is the modern single-stage self-training recipe; we found no reported result on Lacuna for that exact configuration on GTA5→Cityscapes. It shares data, backbone, iterations and a matched tuning budget of an equal number of trials over threshold and loss weights. Beating it by at least 1.5 mIoU on average over three seeds, with gains concentrated on small classes, is our bar.

Semi-supervised domain adaptation. With 100 real labels, the reference is Dual-Level Domain Mixing at 61.2% mIoU (versus ASS at 54.2%) (Dual-Level Domain Mixing). Matching it in one stage, with less compute, counts as success. Exceeding it counts as a clear win.

3D classification. On ModelNet/ShapeNet→ScanNet the reference is Point-Syn2Real at 59.13% (versus DefRec+PCM at 51.80%). We would swap its entropy term for our objective while keeping its sensor simulation fixed.

Ablations. Three ablations isolate the key ingredients. The first uses global rather than domain-relative margins. The second uses randomly chosen negative classes instead of the lowest-ranked ones. The third uses CalibrateMix-style pairing of easy and hard samples driven by raw margins (CalibrateMix) in place of negative labels.

Why not just…?

Why not just lower or adapt the confidence threshold? Lowering the threshold admits more real samples, but it admits them with their top-1 label, which is the quantity the domain gap corrupts. Our first-stage plot measures this directly: if top-1 precision in the low-margin band is acceptable, a lower threshold suffices and our method is unnecessary. A class-adaptive threshold is a real threat, because it partly mimics domain-relative margins. We therefore include a per-class quantile threshold without negative labels as an ablation. If it matches the full method, the contribution reduces to that thresholding and we would report it as such.

Why not just use Maximum Squares Loss on the target? Maximum Squares Loss already rebalances gradient toward hard classes. It still sharpens toward whatever class the model currently favors, so on a real pole the model mistakes for a sign, it strengthens the mistake. Negative labels only push down classes the model is confident are wrong. This difference shows up as small-class IoU and target calibration error. The comparison is run head to head under identical mixing and teacher settings.

Why not just use a frozen foundation backbone and skip the objective? Driving with DINO (Driving with DINO) shows that DINOv3 features, with minor PCA components pruned, carry layout across the CARLA→nuPlan gap. This result is for video generation. It is plausible that a strong pretrained backbone shrinks the discriminative gap enough to erase the benefit of any target loss. We take this seriously and run every baseline and our method on both an ImageNet-initialized and a DINOv3-initialized backbone. Our claim stands only if the negative-label gain survives on the foundation backbone, even if smaller.

Why not just translate synthetic images to look real? PixelDA (Unsupervised Pixel-Level Domain Adaptation with GANs) changes textures and lighting but by its own account cannot fix geometric or sensor differences. Point-Syn2Real found that simulating the sensor mattered more than texture. Translation is complementary to our objective rather than a substitute, so we treat it as an orthogonal input-side add-on, not as a competing baseline.

Decisive Checks

The project lives or dies on three results.

The first is the diagnostic. On real target data, negative labels must be markedly more precise than top-1 pseudo-labels among low-margin samples. They must also stay precise as the gap widens, which we test by comparing GTA5 against the more stylized SYNTHIA.

The second is the ablation against global margins. Domain-relative selection must beat global selection; otherwise the cross-domain argument adds nothing beyond CCL.

The third is backbone robustness. The gain over the strongest practical baseline must persist on the DINOv3 backbone, at three seeds, on both Cityscapes transfers. It must also appear, in matching direction, on ScanNet point clouds, where shortcuts come from sensor geometry rather than texture.

If all three hold, the method gives a simple, single-stage way to learn from the real images that current synthetic-to-real methods discard.