Models trained on video-game imagery and tested on real street scenes fail in a lopsided way. Large classes such as road, sky and building transfer reasonably well. Small, safety-critical classes such as poles, traffic signs, riders and bicycles transfer badly. The usual explanation is class imbalance plus a visual style gap. This note tests a more specific, geometric explanation. On real images, the features for rare classes may fold into the directions already occupied by dominant background classes, so no classifier on top can tell them apart. The representation-learning literature calls this dimensional collapse: features spread across only a few directions of the available space, and the singular values of the feature matrix fall toward zero.
The proposal has two parts. The first is a measurement study. It asks whether this class-level collapse is real in pixel-dense synthetic-to-real transfer (GTA5 or SYNTHIA to Cityscapes), when during training it appears, and whether it predicts per-class accuracy. The second part is a targeted fix that only makes sense if the measurement confirms the story. It is a spread-out penalty applied to the real-domain class centers, with every class weighted equally, instead of to individual pixels. The design matters because pixel-level anti-collapse losses push apart pixels that should stay together, such as two road pixels. The class-center penalty leaves those alone.
Core Claim
The claim has two parts.
The first part is diagnostic. When a segmentation model is adapted from synthetic to real data, the real-domain feature subspace spanned by tail-class pixels comes to overlap increasingly with the subspace of the top few background classes. This overlap, measured by principal angles between the two subspaces and by the effective rank of class-mean features, should predict per-class target IoU better than class pixel frequency alone does.
The second part is interventional. A regularizer that keeps the target-domain class means spread uniformly on the unit sphere should recover tail-class IoU beyond what existing methods achieve. That regularizer uses soft pseudo-labels and counts each class once regardless of its pixel count. Its comparison points are pixel-level uniformity and prototype contrastive alignment.
If the diagnostic correlation fails, the intervention is not worth building, and the negative result is itself informative. It would say that tail failure in synthetic-to-real segmentation is a calibration or frequency problem, not a geometric one.
This extends Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptation, which found that partial domain matching methods collapse target features (singular values dropping sharply) and repaired this with alignment and uniformity losses, uniformity doing most of the work. That work covered whole-image classification only. It also reported that forcing uniformity over everything gives structure to features that ought to be discarded. We move to dense prediction and deliberately restrict uniformity to between-class geometry. This respects the point made in Feature Collapse that contracting within-class variation is constructive: that paper shows early layers mapping equivalent tokens to one direction, and normalization removing magnitude bias from frequency imbalance. Only collapse across classes is destructive.
Research Program
The measurement phase trains a fixed segmentation backbone and decoder under four regimes, each with GTA5 as the labeled source and unlabeled Cityscapes as the target. The regimes are:
- source-only training;
- adversarial feature or output alignment in the style of DANN;
- confidence-thresholded self-training;
- the cross-domain prototype contrastive method of Unsupervised Contrastive Domain Adaptation for Semantic Segmentation.
That paper already observes that hard classes are "absorbed" by dominant background classes. It fixes this by pulling target pixels toward source prototypes and expanding pseudo-labels, but it never measures feature geometry.
At checkpoints throughout training, and at several decoder depths, we use ground-truth Cityscapes validation labels for analysis only, never for training. From them we compute three quantities:
- the singular value spectrum of target pixel features;
- the effective rank of the matrix of the 19 class-mean features;
- the smallest principal angle between each tail class's top-$r$ feature subspace and the span of the road, building, vegetation and sky subspaces.
Effective rank is $\mathrm{erank}(Z)=\exp(-\sum_i p_i\log p_i)$ with $p_i=\sigma_i/\sum_j\sigma_j$, where $\sigma_i$ are the singular values of the feature matrix $Z$. It equals the number of equally used directions.
We also measure the ImageNet-initialized backbone before synthetic training. This separates two possibilities: collapse caused by fitting synthetic data, and collapse inherited from the backbone. This addresses the concern raised in VisDA: The Visual Domain Adaptation Challenge that ImageNet pre-training obscures what adaptation actually does.
The same measurements test a sharp version of the disagreement between DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer and Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation. DIRL argues for adversarial domain invariance, backed by class-conditional constraints and a few real labels. Connect, Not Collapse shows that contrastive features keep domain identity linearly decodable (8% probe error versus 14% for DANN on DomainNet) while transferring better. If adversarial alignment in segmentation produces measurably faster tail-subspace absorption than self-training or contrastive alignment, that is direct evidence in dense prediction for the "disentangle, don't merge" position.
In the intervention phase, we form soft target class means $\mu_k$ in each batch as confidence-weighted averages of normalized target pixel features. The weights are the teacher's softmax probabilities for class $k$. Each mean is maintained as a running average so that rare classes absent from a batch still contribute. The added loss is
$$ \mathcal{L}_{\text{spread}} = \log \frac{1}{K(K-1)} \sum_{j \neq k} \exp\!\left(-t\,\lVert \hat\mu_j - \hat\mu_k \rVert^2\right), $$
where $\hat\mu_k$ is the normalized mean of class $k$, $K=19$ is the number of classes and $t$ is a temperature. This is the uniformity loss of Fang et al. applied to $K$ class means instead of thousands of pixels. Because each class enters exactly once, the gradient on "pole" is as large as the gradient on "road." Within-class pixels are not pushed apart at all.
The loss is added on top of the self-training baseline and, separately, on top of the prototype contrastive method. This tells us whether it adds anything beyond what contrastive separation already provides.
Baselines
The naive reference point is source-only training on GTA5, evaluated on Cityscapes. VisDA: The Visual Domain Adaptation Challenge reports 21.4 mIoU for its source-only segmentation baseline on the VisDA-S track, and 47.5 mIoU for its adversarial challenge winner. Both used older architectures. We will rerun source-only with our backbone rather than quote these figures as directly comparable.
The strongest existing method is cross-domain prototype contrastive adaptation from Unsupervised Contrastive Domain Adaptation for Semantic Segmentation. It reports 60.2% mIoU on Cityscapes validation from GTA5 and 56.5% from SYNTHIA, with gains above 10% on traffic light, sign and bicycle. It is the strongest competitor because it already targets tail-class absorption with an explicit separation mechanism. We will reimplement it in our codebase with the same backbone, crop size, iteration budget and pseudo-label threshold schedule. We will first confirm that our reproduction lands near 60.2 on GTA5 to Cityscapes, and report the reproduced figure. Beating it means a gain in mean IoU over the eight rarest classes that exceeds three-seed standard deviation, with no loss in overall mIoU, on both source datasets.
The key-ingredient ablations are two. The first is pixel-level uniformity: the Fang et al. loss applied directly to sampled target pixel features, which isolates whether the between-class restriction matters. The second is frequency-weighted class-mean spread, where each class pair is weighted by pixel count, which isolates whether equal class weighting is what helps tails. All variants get the same hyperparameter search budget over loss weight and temperature. Fang et al. note that their loss weight was stable only between 0.3 and 0.7, so we sweep an equal grid for every method.
Why not just…?
Why not just apply Fang et al.'s uniformity loss to pixels? This is the obvious transplant, and it is included as an ablation. There is a reason to expect it to hurt. Uniformity over pixels treats millions of road pixels as things to spread apart, which fights the within-class contraction that Feature Collapse argues is useful. Fang et al. themselves note side effects from imposing structure on features that should be discarded. If pixel uniformity matches our class-mean version on tail IoU, the restriction is unnecessary and we will say so.
Why not just rebalance classes with rare-class sampling or a class-balanced loss? This is the most serious threat. If tail failure is purely a frequency effect, reweighting fixes it with no geometry involved. The diagnostic phase settles this by testing whether subspace overlap predicts per-class IoU after controlling for pixel frequency. The intervention is also compared against class-balanced self-training given the same tuning budget. If class balancing alone closes the tail gap and the geometric signal disappears once frequency is controlled for, the proposal's premise is wrong.
Why not just rely on prototype contrastive alignment, which already separates classes? That method pulls target pixels toward source prototypes. It does not constrain how target class means relate to each other, and its pixel-sampled contrastive terms are still dominated by frequent classes. Whether this leaves residual tail absorption is an empirical question. Stacking our loss on top of it answers the question directly, and a null result there would limit the contribution to the diagnostic study.
Why not just render more synthetic data? Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks? shows that scaling GTA V data to 200k images beats real Cityscapes training for vehicle detection (0.6803 versus 0.6247 mAP on KITTI Easy). Vehicles, however, are a well-represented class, and the VisDA source-only numbers show that tens of thousands of synthetic frames still transfer poorly. We will measure tail-subspace overlap at two source-set sizes to see whether scale alone reduces it.
Decisive Checks
The first check is the diagnostic correlation, run across the four training regimes, three seeds and 19 classes on GTA5 to Cityscapes. A partial correlation between tail-subspace angle and per-class IoU that stays clearly nonzero after accounting for class frequency supports the geometric story. A partial correlation near zero ends the intervention track.
The second check is the ImageNet-backbone measurement. It shows whether absorption is caused by synthetic fine-tuning. If it is, the absorption should be absent or weaker at initialization and grow during source training.
The third check is the intervention comparison on both GTA5 and SYNTHIA to Cityscapes. It reports overall mIoU, rare-class mIoU, and class-mean effective rank. The loss must raise both rank and rare-class IoU. A gain in rank without a gain in IoU would mean we are regularizing the wrong quantity.
As a sanity replication, we will run the class-mean spread loss on VisDA-C classification to confirm that it behaves no worse than Fang et al.'s loss where theirs was designed to work. No reported result for Fang et al.'s method on VisDA was found on Lacuna, so we will generate that comparison ourselves.
References
- Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptation
- Feature Collapse
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation
- VisDA: The Visual Domain Adaptation Challenge
- DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer
- Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?