The best methods for adapting a segmentation model from game-engine images to real street scenes work in a particular way. A "student" network is trained on real, unlabeled images using pseudo-labels: guessed labels produced by a "teacher" that is simply a slowly updated running average of the student itself. Because this teacher is a copy of the student, it inherits the same blind spots. When the student has learned a simulator shortcut, such as a rendering texture or a fixed lighting cue, the teacher confidently confirms the resulting mistakes on real images, and the error reinforces itself. We propose replacing this self-copy with a partner. Two peer networks are trained together, and each is deliberately blocked from a different synthetic shortcut: one cannot rely on texture and appearance, the other cannot rely on scene context. On real images, each peer is taught by the other's averaged copy rather than its own. The hypothesis is that pseudo-labels improve to the extent that the teacher's errors on real data are uncorrelated with the student's, and that blocking different shortcuts produces this decorrelation more reliably than random seeds do.
This addresses both research directions at once. On the teacher-dependency side, it removes the need for an external frozen teacher or a stronger model. That includes the ImageNet feature-distance anchor that DAFormer adds to keep self-training stable. On the synthetic-to-real side, it targets the documented failure of the running-average teacher, and it is designed to help most where context-based fixes are weakest, such as sparse LiDAR scans.
Core Claim
Self-training across the synthetic-to-real gap is limited by how correlated the errors of the pseudo-label source and the learner are, not by how strong the teacher is. Two peers whose source-domain training removes different shortcut cues will make less correlated errors on real data than a student and its own running average. Cross-supervising them should therefore beat both self-averaging and symmetric mutual learning, at the same training compute, while only one peer is deployed at test time.
The core objective for peer $A$ on a target image $x_t$ is
$$ \mathcal{L}_A = \mathcal{L}_{\text{src}}(f_A; \tau_A) + \lambda\, \mathcal{L}_{\text{ce}}\big(f_A(m(x_t)),\ \hat y_t^{\bar B}\big) + \mu\, \mathrm{KL}\big(f_B(x_t)\,\|\,f_A(x_t)\big). $$
The terms are defined as follows. $f_A$ and $f_B$ are the two peers. $\tau_A$ is the shortcut-blocking corruption applied only to $A$'s synthetic inputs. $m$ is the target masking used in MIC. $\hat y_t^{\bar B}$ is the confidence-weighted pseudo-label produced by the running average $\bar B$ of peer $B$. $\lambda$ and $\mu$ are loss weights. Peer $B$'s loss is defined symmetrically, with its own corruption $\tau_B$.
The diagnostic we track throughout training is the error correlation $\rho$ between the pseudo-label generator and the learner. We compute it as the phi coefficient between their per-pixel error indicators on held-out labeled real images, which are used only for analysis.
How This Sits Against Prior Work
Two papers motivate the problem. DAFormer builds a transformer segmenter trained with running-average self-training. It reports that naive self-training collapses toward dominant classes such as "road," and it needs rare-class sampling, warmup, and feature-distance distillation from ImageNet features to stay stable. MIC adds a task in which the student predicts the teacher's labels from a 70%-masked image. That forces the student to use context, but the supervision source is still a running average of the student itself. We keep MIC's masking and change only where the pseudo-labels come from. We also test whether the ImageNet feature-distance term becomes unnecessary.
Two further papers supply the method. Deep Mutual Learning trains peer students with symmetric KL mimicry. On clean CIFAR-100, it shows that this can beat distillation from a large teacher: MobileNet reaches 76.13% with peers versus 69.12% distilled from a Wide ResNet. That work had no domain shift and no deliberate asymmetry between peers. Our central ablation asks whether symmetric peers are enough. Adversarially Masking Synthetic to Mimic Real learns point-dropout masks that make synthetic LiDAR resemble real scans. We reuse it as one peer's shortcut-blocking corruption on the LiDAR benchmark.
Feature Distillation Improves Zero-Shot Transfer from Synthetic Images constrains the design. It shows that matching a frozen CLIP image encoder shields a student trained on synthetic images from spurious features. That is strong evidence for teacher-based shortcut protection, but it requires the large teacher and covers classification only. We treat it as a competitor rather than a component.
The robotics privileged-teacher methods, Learn to Teach, HIB, and TWIST, all depend on simulator state that has no counterpart for real camera images. We do not build on them, although TWIST names co-evolving, teacher-free training as an open direction.
Peer-based pseudo-labeling for domain adaptation may well exist outside this research file. The file does not contain it. What we claim as new is narrower: shortcut asymmetry is the source of peer diversity, and error decorrelation is the measured explanatory variable.
Research Program
We build on the official MIC codebase with the DAFormer architecture (MiT-B5 encoder) for images. For LiDAR, we use the setup from the adaptive-masking paper.
For images, peer $A$ is the appearance-blind peer. Its synthetic inputs receive strong photometric and style randomization together with texture-destroying augmentation, so it must rely on shape and layout. Peer $B$ is the context-blind peer. Its synthetic inputs receive heavy random patch masking, and it is trained on small crops, so it must rely on local appearance. On real images, both peers receive the same MIC masking and are supervised by each other's running averages.
For LiDAR, peer $A$ trains on synthetic scans corrupted by learned adaptive masking. Peer $B$ trains on synthetic scans with random beam subsampling and density thinning, which matches sensor sparsity without the adversarial component.
The analysis tracks $\rho$ over training for each pseudo-label source. If error correlation is the operative variable, then the per-class pseudo-label accuracy gain over the self-average baseline should be predicted by the per-class drop in $\rho$. In particular, rare classes such as "train" and "truck," which DAFormer shows are where confirmation bias bites hardest, should show both the largest drop in $\rho$ and the largest gain.
At deployment we keep a single peer. We pick it on a labeled source validation split, never on target labels, so that inference cost matches the baselines.
Baselines
The naive reference is the source-only model: the same architecture trained on GTA5 or SynLiDAR and evaluated directly on the real domain. On SynLiDAR→SemKITTI, the adaptive-masking paper reports 15.2% mIoU for this setting, against a 55.5% mIoU real-supervised oracle. This defines the gap we are trying to close.
The strongest existing image method is MIC on top of DAFormer, which reports 75.9 mIoU on GTA5→Cityscapes (MIC). DAFormer alone reports 68.3 mIoU (DAFormer). Running two peers doubles training compute, so we will run MIC under two compute-matched variants: twice the iterations, and two independently seeded MIC runs whose running-average teachers are averaged into an ensemble. Hyperparameter search budgets will be equal across methods. Beating MIC means a single deployed peer that exceeds our own reproduced MIC number under both matched variants, averaged over three seeds with non-overlapping spread. If the reproduced MIC figure falls well short of 75.9, we report that and compare against the reproduction.
For LiDAR, the strongest reported method is adaptive spatial masking. It reports 22.8% mIoU on SynLiDAR→SemKITTI and 17.0% on SynLiDAR→nuScenes, versus 20.0% for CoSMix (Adversarially Masking Synthetic to Mimic Real). Here we expect the clearest margin, because MIC's own limitations note that context-based consistency helps little when context is uninformative.
Two ablations isolate the key ingredient. The first is symmetric mutual learning: two peers with identical augmentation, cross-supervised in the same way, as in Deep Mutual Learning. This tests whether peers alone help, or whether shortcut asymmetry is required. The second is self-supervision with asymmetric augmentation: each peer is trained with its own corruption but supervised by its own running average. This separates the effect of the corruption from the effect of who provides the pseudo-labels. A third check removes DAFormer's ImageNet feature-distance term from both our method and MIC, which measures how much each method depends on that external anchor.
Why not just…?
Why not just ensemble two seeded MIC runs? This is the real threat. If seed diversity already decorrelates errors, our method reduces to an ensemble with extra steps. The compute-matched seed-ensemble baseline settles the question directly, and $\rho$ shows the mechanism. We predict that seeded copies keep high error correlation because they learn the same synthetic shortcuts. If their $\rho$ and mIoU match ours, the shortcut-asymmetry claim is wrong and we will say so.
Why not just apply both corruptions to a single student? Combining texture randomization and masking in one MIC model is cheaper. The "self-supervision with asymmetric augmentation" ablation, extended to a single model that receives both corruptions, tests this. A single model can still settle on whichever shortcut survives both corruptions, and its running average shares that choice. If the single combined model matches the peers, the benefit came from augmentation rather than from cross-supervision.
Why not just distill from a frozen foundation model on the real images? Feature Distillation Improves Zero-Shot Transfer from Synthetic Images shows that a frozen CLIP encoder protects a student against synthetic spurious features. That evidence covers classification, though, and it requires hosting a large teacher, which is exactly the dependency this work aims to remove. The file reports no dense-segmentation or LiDAR result for that approach. We include a CLIP feature-distance variant in place of DAFormer's ImageNet term as a reference point. If it dominates, the honest conclusion is that teacher-free peers are a cheaper second-best, not a replacement.
Why not just generate more synthetic data? Driving in the Matrix shows that scaling GTA V data to 200k frames beat real Cityscapes training on KITTI vehicle detection (0.6803 vs 0.6247 mAP on Easy). Yet the large SynLiDAR set yields only 15.2% source-only mIoU on SemKITTI, because sensor artifacts are missing from the generator. Scaling does not remove shortcuts the simulator never varies. We still report source-only and MIC results at a 50% source subset to check whether our gains shrink as data grows.
Proposed Experiments
The primary benchmarks are GTA5→Cityscapes and Synthia→Cityscapes for images, and SynLiDAR→SemKITTI and SynLiDAR→nuScenes for LiDAR, evaluated by mIoU with per-class IoU. For Synthia→Cityscapes and DarkZurich, the file reports only relative gains (+5.4 for DAFormer, +4.3 for MIC), so no absolute reported result was found on Lacuna for either. We will therefore compare against our own reproductions there. DarkZurich serves as a stress test for appearance shift. VisDA-2017, where MIC reports 92.8%, is a cheap classification sanity check.
The decisive check has three parts, all of which must hold. First, our method beats the compute-matched seeded MIC ensemble on GTA5→Cityscapes. Second, it beats symmetric mutual learning. Third, across classes and training time, its gains track lower $\rho$. If gains appear without lower $\rho$, the mechanism is something other than decorrelation. If lower $\rho$ appears without gains, decorrelation is not the bottleneck.
A secondary check expects the margin over the strongest baseline to be larger on SynLiDAR→SemKITTI, relative to the 22.8% reported for adaptive masking, than on GTA5→Cityscapes. That pattern would support the claim that peer cross-supervision helps most where context is weak. The final check measures how much mIoU each method loses when the ImageNet feature-distance term is removed. A smaller drop for our method would show it relies less on that external anchor.
References
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation
- Deep Mutual Learning
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Feature Distillation Improves Zero-Shot Transfer from Synthetic Images
- Learn to Teach: Improve Sample Efficiency in Teacher-student Learning for Sim-to-Real Transfer
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?