A segmentation network trained on video-game or simulator images often fails on real street photos. It learns to recognize "road" from the simulator's particular textures and colors rather than from the shape and layout of the scene. Simulators, however, hand out geometric labels for free: per-pixel depth, and surface orientation derived from that depth. These labels describe properties of a scene that do not change when rendering style changes. This note proposes a simple, testable way to use them. Train a shared encoder to predict depth, surface normals and semantic boundaries through a small shared feature space. Then force the segmentation head to read only from that geometry-shaped space. The segmentation loss is allowed to shape the encoder, but it is not allowed to open new feature directions that the geometry tasks do not use.
The central claim is falsifiable. Simply attaching auxiliary geometry heads to a shared encoder (standard multi-task learning) should give little out-of-domain benefit, because the segmentation head can still find its answers in feature directions the geometry heads never constrain. Routing segmentation through a geometry-owned bottleneck is the ingredient that turns free simulator geometry into robustness on real target cities that the model never sees during training. If plain auxiliary heads turn out to match the bottleneck, that result is itself informative. It would be the first clean source-only measurement of how much auxiliary geometry buys in domain generalization. The design below is built so either outcome is reported cleanly.
What exists and what is missing
The closest work is DADA: Depth-Aware Domain Adaptation in Semantic Segmentation. It adds a depth branch to a segmentation network trained on SYNTHIA and multiplies depth features into appearance features. It then uses predicted depth to weight an adversarial discriminator, reaching 42.6% mIoU on SYNTHIA→Cityscapes. That discriminator needs unlabeled target images, so DADA solves domain adaptation (target images available during training) rather than domain generalization (target never seen). Its depth head mostly serves as a weighting mask for alignment, not as a constraint on what segmentation may use.
Hard Pixel Mining for Depth Privileged Semantic Segmentation uses training-only depth to reweight hard pixels. It gains +3.06% IoU over a plain segmentation-plus-depth multi-task baseline on NYUDv2, but only in-domain. Joint 2D-3D Multi-Task Learning on Cityscapes-3D shows joint depth, segmentation and 3D detection helping each other (3D detection 26.90→32.94 mDS), again only within Cityscapes. These three papers motivate the proposal: geometry demonstrably shapes segmentation features, but nobody has tested whether it does so in a way that survives a synthetic-to-real shift without target data.
On the domain-generalization side, Augmentation-based Domain Generalization for Semantic Segmentation runs a full factorial study of input augmentations. Elastic transforms and random resized crops matter most, reaching 39.5% mIoU (ResNet-101) and 44.2% (DAFormer) on Synthia→Cityscapes, and some augmentations interfere with each other when combined. Augment Features Beyond Color for Domain Generalized Segmentation hallucinates styles in feature space with a GAN, reaching 45.30% average mIoU from GTAV to three real datasets. Domain Generalization for Semantic Segmentation: A Survey treats multi-task learning as orthogonal to domain generalization. It also records two points this proposal relies on. Domain-invariant shape sits mainly in high spatial frequencies and style in low ones. And many "domain generalization" methods quietly import real-image priors. These works supply the baselines and the evaluation protocol.
Two further papers constrain the design. In Search of Forgotten Domain Generalization shows that CLIP trained only on natural images loses about 60% of its relative accuracy on renditions. Apparent foundation-model robustness is therefore largely exposure to the test domain, so we cannot assume a large pretrained backbone already solves the problem. The theory in Optimal criterion for feature learning of two-layer linear neural network in high dimensional interpolation regime proves, for linear two-layer networks, that pooling many outputs recovers a shared latent basis that a single output cannot. It gives the method its shape: the "shared basis" should be owned by the tasks whose labels are style-invariant.
Core Claim
Let $h(x)$ be encoder features at each pixel of image $x$. Let $P$ be a learned projection to a rank-$r$ space with $r$ much smaller than the feature width. Let $g_k$ be heads for depth, surface normals and semantic boundaries, and let $s$ be the segmentation head. All heads read from $z = P\,h(x)$. The training objective is
$$ \mathcal{L} = \sum_k \lambda_k \,\ell_k\big(g_k(P h(x)),\, y_k\big) + \ell_{\text{seg}}\big(s(\mathrm{sg}[P]\, h(x)),\, y_{\text{seg}}\big), $$
where $y_k$ are simulator geometry labels, $\lambda_k$ are task weights, and $\mathrm{sg}[\cdot]$ is stop-gradient. The segmentation loss updates $s$ and the encoder, but it cannot rotate $P$. Segmentation therefore only sees information lying in directions the geometry tasks chose.
The encoder can still try to write texture information into those directions. The geometry losses push back on this, because texture does not predict depth once the photometric augmentation is applied, and depth labels are unaffected by that augmentation. The claim is that this construction produces a larger source-only gain on real targets than plain auxiliary heads do. It also predicts a measurable mechanism: segmentation accuracy should depend less on low-frequency image content.
Research Program
The program has three parts. The first is building the model. The primary source domain is SYNTHIA, because it ships per-pixel depth, which is why DADA used it. Normals are computed from depth and camera intrinsics, and semantic boundaries come from the segmentation labels, so no new labels are needed. The backbones are ResNet-101 with DeepLab-style decoding and the DAFormer architecture, matching the reported numbers in the augmentation study. All variants share the same elastic-transform and random-resized-crop augmentation, so the multi-task effect is measured on top of the best known cheap recipe rather than instead of it.
The second part is the mechanism study. We filter test images into low-pass and high-pass versions and measure the segmentation accuracy retained under each. This directly tests whether the bottleneck shifts reliance toward the high-frequency structure the survey links to domain invariance. We also track per-head gradient norms reaching the encoder during training. RIFLE showed that a fast-converging head starves early layers of gradient. We expect segmentation on easy synthetic data to converge fast while geometry heads keep sending gradient, and we check whether the bottleneck's benefit tracks this.
The third part is portability. We apply the same bottleneck while fine-tuning a large pretrained backbone. This asks whether geometric routing still adds anything once a strong prior is present, or whether it only helps weak backbones.
Baselines
The naive reference is a single-task segmentation network trained on SYNTHIA with elastic transform and random resized crop. Augmentation-based Domain Generalization for Semantic Segmentation reports 39.5% mIoU (ResNet-101) and 44.2% mIoU (DAFormer) on Synthia→Cityscapes for this setup. We rerun it in our codebase to confirm we reproduce these numbers before comparing anything else.
The strongest competitor on the precise question is plain multi-task learning: the same encoder, the same augmentation, and depth, normal and boundary heads attached directly to $h$ with no bottleneck. It is the obvious reading of the multi-task-helps-generalization idea, and the proposal stands or falls on beating it. No reported result was found on Lacuna for source-only multi-task geometry on Synthia→Cityscapes. We also run a DADA-style variant that fuses depth features into appearance features but drops the discriminator, making it source-only. And we run a source-only port of the hard-pixel loss weighting from Hard Pixel Mining.
The strongest existing domain-generalization method in the file is GBFA from Augment Features Beyond Color. It reports 45.30% average mIoU on GTAV→{Cityscapes, BDDS, Mapillary} with ResNet-50, compared with 42.42% for SHADE. We rerun GBFA on our SYNTHIA source with its published schedule, and we also run our method with GBFA's GTAV protocol for the comparison. GTAV does not supply depth by default, so that comparison uses depth only where it is available. If it is not available, we report that the GTAV comparison is limited to boundary supervision.
As an out-of-reach reference, SoRA's 68.3% and MFuser's above-71% mIoU on GTA5→Cityscapes, both reported in the survey, mark what large pretrained models achieve. We do not claim to beat these with ResNets.
For fairness, every method gets the same source data and augmentation, the same iteration budget, and the same hyperparameter search budget (an equal number of trials tuned on a held-out SYNTHIA split, never on target data). Results are reported as mean and spread over three seeds.
Three ablations isolate the key ingredient. The first lets segmentation gradients flow into $P$, which tests whether the stop-gradient matters. The second keeps the rank-$r$ bottleneck but removes the geometry losses, which tests whether a plain bottleneck regularizes on its own. The third replaces geometry targets with random smooth fields of matched statistics, which tests whether any dense auxiliary target works equally well. We also sweep $r$.
Beating plain multi-task learning means a gain of at least two mIoU points averaged over the three real targets, larger than the seed spread, with the bottleneck-only and random-target ablations recovering less than half of that gain.
Why not just…?
Why not just use stronger augmentation on a single-task model? Augmentation is cheap and the augmentation study shows it is strong. The same study also shows diminishing returns and negative interactions when transforms are stacked, which suggests input-level randomization saturates. We do not argue against augmentation; every variant includes the best recipe. The question is whether geometric routing adds on top. The single-task baseline directly measures this.
Why not just add auxiliary depth and normal heads without the bottleneck? This is the real threat. Hard Pixel Mining's plain multi-task baseline already improved in-domain segmentation, and plain sharing may carry over to the target domain. Our hypothesis is that it transfers weakly, because segmentation can rely on directions geometry never constrains. The plain multi-task baseline and the stop-gradient ablation settle this directly. If plain heads match the bottleneck, we report auxiliary geometry as the finding and drop the routing claim.
Why not just use a large pretrained backbone? The forgotten-domain-generalization study shows that such robustness largely reflects training data that already covers the test style. SYNTHIA→real driving is plausibly well covered by web data, so a foundation model will win on raw numbers. The relevant question is whether routing still adds a gain there. The portability experiment tests exactly this. A null result would mean the method matters mainly when strong priors are unavailable, as in the "proxy data" setting the survey warns about.
Why not just use a low-rank bottleneck, which is a generic regularizer? A narrow feature space could improve generalization simply by limiting capacity. The bottleneck-without-geometry ablation and the random-target ablation separate "narrow" from "geometry-shaped." Only the second is the claim.
Proposed Experiments
We train on SYNTHIA and evaluate zero-shot on Cityscapes, BDD and Mapillary, reporting per-target and averaged mIoU plus per-class results for thin and near-field classes, where DADA saw its largest gains. A secondary run uses GTAV as source with boundary and any available depth supervision, for comparison with GBFA's 45.30%.
The first decisive check is whether the bottleneck beats plain multi-task learning by the stated margin while the ablations do not. The second is whether the frequency test shows the bottleneck model retaining more accuracy on high-pass images and losing less on low-pass images than both single-task and plain multi-task models. The third is whether the benefit survives with a DAFormer backbone and the pretrained-backbone fine-tuning.
If all three hold, the result is concrete: simulator geometry is useful for real-world generalization mainly when segmentation is forced to see the world through it. If only plain auxiliary heads help, the study still provides the missing source-only measurement that DADA, Hard Pixel Mining and TaskPrompter left open.
References
- DADA: Depth-aware Domain Adaptation in Semantic Segmentation
- Hard Pixel Mining for Depth Privileged Semantic Segmentation
- Joint 2D-3D Multi-Task Learning on Cityscapes-3D: 3D Detection, Segmentation, and Depth Estimation
- Augmentation-based Domain Generalization for Semantic Segmentation
- Augment Features Beyond Color for Domain Generalized Segmentation
- Domain Generalization for Semantic Segmentation: A Survey
- In Search of Forgotten Domain Generalization
- Optimal criterion for feature learning of two-layer linear neural network in high dimensional interpolation regime
- RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr