Models trained on rendered images, such as CAD objects or video-game street scenes, collapse on real photographs. One benchmark records an AlexNet classifier falling from 99% on synthetic CAD images to 28% on real COCO images. The usual remedy changes the pixels: networks that translate synthetic images to look real, and discriminators that force feature statistics to match. A separate result from dataset distillation points another way. When a strong teacher supplies soft labels (full probability vectors over classes rather than a single correct class), the choice of training images matters surprisingly little. Random real images with expert soft labels beat carefully synthesized images.
This note asks whether that "labels over pixels" result survives a domain gap. The procedure is simple. Train a teacher on labeled simulator data. Use it to write soft labels onto many augmented crops of unlabeled real target images. Train a student only on those relabeled real crops, with no image translation and no adversarial alignment. The open question is which parts of a simulator-trained teacher's probability vector are worth keeping. Its top choices may carry real semantics, while its small "tail" probabilities may mostly encode rendering shortcuts. We propose to measure how many rendered labeled images one relabeled real image is worth. We also propose a cheap, continuous way to discount the parts of the teacher's distribution that come from rendering artifacts.
Core Claim
Our hypothesis is that once a simulator-trained teacher's soft labels are moved onto real target images, most of the remaining sim-to-real error lies in specific, identifiable parts of the teacher's distribution, not in image appearance. If so, offline relabeling of real crops should close much of the gap that heavy pixel-translation pipelines close, at a fraction of the compute. The useful signal should sit in class pairs whose confusion is stable when rendering style changes, such as truck versus car. The harmful signal should sit in class pairs whose confusion appears or disappears with rendering style.
The method has one new ingredient: stability-weighted soft relabeling. Using only labeled synthetic data, we estimate for each true class $y$ and other class $j$ a stability score $s_{yj} \in [0,1]$. The score is one minus the normalized variation of the teacher's probability for $j$ across style-randomized versions of the same synthetic image, covering texture, lighting, blur and sensor noise. On a real crop $x$, let $p_j(x)$ be the teacher's probability for class $j$ and $\hat{y}$ its top class. The student is then trained toward
$$ q_j(x) \propto \begin{cases} p_{\hat y}(x) & j=\hat y \\ p_j(x)\, s_{\hat y j}^{\gamma} & j\neq\hat y \end{cases} $$
where $\gamma \ge 0$ sets how hard unstable confusions are suppressed. At $\gamma=0$ this is plain distillation. As $\gamma$ grows it approaches a learned, class-pair-specific truncation.
This differs from the closest work in three ways. Probabilistic Filtered Soft Labels for Domain Adaptation (IFCDA) collapses confident target labels to one-hot and truncates the rest to the top 2–4 classes. Those labels come from graph propagation, and the method was tested only on real-to-real Office/Caltech benchmarks. We keep the distribution continuous, use a learned teacher, and decide what to suppress from measured style sensitivity rather than a fixed rank cutoff. Squeeze, Recover and Relabel (SRe²L) and A Label is Worth A Thousand Images in Dataset Distillation relabel crops offline, but always within one closed domain. We reuse their relabeling machinery across a domain boundary, where the teacher has never seen a real image.
Research Program
The program has three parts. The first two answer factual questions; the third tests the method.
The first part measures an exchange rate. On VisDA-C (12 classes; about 152k synthetic CAD renders and about 55k real COCO validation crops, per VisDA), we train students on grids of two quantities. One is $N_s$, the number of synthetic images with ground-truth hard labels. The other is $N_r$, the number of unlabeled real images relabeled by the simulator teacher. We fit curves of equal accuracy across this grid. Qin et al. report that soft labels give a roughly constant 6x data reduction over hard labels within ImageNet. We ask whether a comparable constant holds across the gap, and whether it shrinks as the teacher gets worse. All COCO images used for relabeling are kept disjoint from the evaluation images.
The second part characterizes the teacher. Qin et al. found that early-stopped, higher-entropy teachers can be better soft-label sources. Work on jet tagging, by Bauer et al. and Kagan et al., shows that networks latch onto simulator-code artifacts. These two findings predict opposite things for early teachers in simulation. We relabel the same real crops with checkpoints from across the teacher's training and with several temperatures. For each, we record three quantities: label entropy on real data, the student's real accuracy, and a shortcut index, defined as the mean of $1-s_{yj}$ weighted by the teacher's off-target mass. We also replicate the comparison of label-smoothed and hard-label teachers from Is Label Smoothing Truly Incompatible with Knowledge Distillation, this time under domain shift.
The third part tests stability-weighted relabeling on classification and on dense segmentation. For segmentation, we move to GTA5→Cityscapes and to GTA5+SYNTHIA→Cityscapes/BDDS. Here soft labels are per-pixel distributions. Storage becomes a real cost, a limitation the file flags for multi-crop relabeling. We will store labels at reduced spatial resolution with 8-bit quantization and report the storage cost alongside accuracy.
Segmentation is where the long-tail risk shows up. Shen et al. report that soft targets hurt rare classes on long-tailed data. Multi-source Domain Adaptation for Semantic Segmentation (MADAN) reports that thin classes such as poles, signs and fences remain its weakest. Per-class IoU on these classes is therefore a primary metric, not an afterthought.
For the multi-source setting, we form the soft label from the two simulator-specific teachers, combined by geometric mean before stability weighting. This tests whether combining the sources in label space can replace MADAN's pixel-level aggregation discriminators.
Baselines
The naive reference is a source-only student: trained on synthetic images with hard labels, evaluated on real data. VisDA reports 28% accuracy on VisDA-C (AlexNet) and 21.4 mIoU on GTA5→Cityscapes. MADAN reports 25.8 mIoU when GTA5 and SYNTHIA are combined naively. We rerun all of these on our shared backbone, because the published numbers mix architectures.
The strongest classification competitor is hard pseudo-label self-training with confidence thresholds and a Mean Teacher, the family that won VisDA-C at 92.8% accuracy (reported in VisDA). The strongest segmentation competitors are pixel translation plus adversarial alignment. The VisDA-S winner reached 47.5 mIoU on Cityscapes. In the multi-source setting, MADAN reached 41.4 mIoU on Cityscapes and 36.3 on BDDS, run from the released luodian/MADAN code.
To keep comparisons fair, every method uses the same backbone, the same unlabeled real images, and the same augmentation family. Every method gets equal hyperparameter-search trials, following the standardized evaluation argument of Rectified Decoupled Dataset Distillation, which showed that protocol differences inflated gaps between methods from 6.7% to 27.3%. We report GPU-hours for each method, so that "matching MADAN at a fraction of the compute" is a measured claim.
The proposal beats a baseline only if it meets three conditions at once. It must match or exceed the baseline's accuracy or mIoU on the shared backbone. Its gains must hold across three seeds. It must not lose thin-class IoU.
Ablations isolate the stability weight. The first is plain full-distribution distillation ($\gamma=0$). The second is IFCDA-style top-k truncation with hard collapse of confident labels. The third is temperature flattening. The fourth is static label smoothing. The fifth is averaging the teacher over test-time augmentations without stability weights, which is the most likely cheaper explanation for any gain. A final arm feeds the stability-weighted labels into the hard-pseudo-label self-training loop, to test whether the two are complementary rather than competing.
Why not just…?
Why not just use hard pseudo-labels with a confidence threshold, as the VisDA winner did? This is the strongest practical competitor, and it may win on overall accuracy. Our claim is narrower and checkable. On ambiguous boundary samples and thin classes, hard labels throw away exactly the stable confusion structure we keep. IFCDA's motivation that overconfident hard labels cause negative transfer supports this, though only on real-to-real data. If stability-weighted soft labels add nothing on top of thresholded self-training in the combined arm, we will report that relabeling is redundant with it.
Why not just translate the pixels, with CycleGAN-style pipelines such as MADAN? MADAN's gains are real (25.8→41.4 mIoU), but they come with several discriminators, added training cost, and admitted failures on thin structures. The exchange-rate experiment and the GPU-hour accounting settle the question directly. If relabeled real frames reach MADAN's mIoU with no generator, pixel translation is unnecessary for this purpose. If they fall far short, the "labels over pixels" finding does not cross the gap, and that negative result is itself worth publishing.
Why not just skip the simulator and relabel real images with a teacher trained on real web data? This is a genuine threat for VisDA's everyday classes, and Qin et al.'s random-real-plus-expert result suggests it could be strong. We have no reported result found on Lacuna for such a teacher on VisDA-C or Cityscapes. We will run it as a baseline, along with a combined teacher that averages it with the simulator teacher. If the combined teacher does not beat it, the simulator's labels add nothing here. That would restrict our claim to domains with no real-data teacher, such as the spacecraft imagery in Annotation Free Spacecraft Detection or crash scenes.
Why not just use label smoothing, which Shen et al. show shares the distillation objective? Smoothing spreads probability mass uniformly, so it cannot tell a semantic confusion from a shortcut confusion. It is also reported to shrink weights on rare classes, which are exactly the classes where sim-to-real transfer is weakest. It is included as an ablation arm, and per-class thin-object IoU will show whether it hurts the tail.
Proposed Experiments and Decisive Checks
The first decisive check is the VisDA-C exchange-rate grid. Labels-over-pixels holds across the gap if a student trained only on relabeled real crops beats the source-only student trained on all 152k synthetic hard-labeled images while using far fewer real images. The second check concerns the shortcut index. Across teacher checkpoints and temperatures, the index should predict student real-data accuracy better than label entropy does. If entropy predicts better, the stability score is not capturing what matters.
The third check is the head-to-head comparison on GTA5→Cityscapes and GTA5+SYNTHIA→Cityscapes/BDDS against MADAN and thresholded self-training. We report mean IoU, IoU on poles, signs and fences, storage, and GPU-hours. The method fails its central claim in either of two cases: the test-time-augmentation ensemble matches it, or it falls more than a few points short of MADAN while not reducing compute substantially. In either case we report the clean measurement of how far relabeling alone carries across the sim-to-real gap.
References
- Probabilistic Filtered Soft Labels for Domain Adaptation
- Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective
- A Label is Worth A Thousand Images in Dataset Distillation
- VisDA: The Visual Domain Adaptation Challenge
- Jet substructure classification in high-energy physics with deep neural networks
- Boosted Jet Tagging with Jet-Images and Deep Neural Networks
- Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
- Multi-source Domain Adaptation for Semantic Segmentation
- Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation
- Annotation Free Spacecraft Detection and Segmentation using Vision Language Models