Models trained on video-game or simulator imagery break on real photographs. Part of the reason is that they latch onto rendering quirks, such as overly clean textures and deterministic shading, that never appear in real cameras. This proposal tests a simple hypothesis. Inside a segmentation network's features, those simulator quirks occupy a small set of directions. If so, a filter that projects features onto the remaining directions should remove the shortcut while keeping the semantics. The filter is an orthonormal matrix, meaning a matrix whose columns are unit-length and mutually perpendicular. It is trained jointly with the network, and the rule is that it may only choose which directions to keep. It may not rescale them.
The orthonormality constraint is the core of the idea, not decoration. An ordinary learned linear map asked to make synthetic and real features indistinguishable has an easy way out: it shrinks or blurs feature dimensions until nothing is distinguishable, semantics included. A matrix with orthonormal columns has no such freedom. To satisfy the domain objective it must rotate the kept subspace away from simulator-specific directions. The set of such matrices is called the Stiefel manifold, and training on it requires Riemannian optimization: gradient steps that stay on the curved set of orthonormal matrices rather than leaving it and being forced back. The bet is that this "select, don't squash" constraint closes part of the remaining synthetic-to-real gap on top of the strongest self-training pipelines. A second payoff is an inspectable object: the discarded subspace, which can be probed directly to see what the simulator shortcut looks like.
Core Claim
The claim has three parts. A learnable orthonormal filter $W \in \mathrm{St}(p,d)$, inserted between encoder and decoder of a domain-adaptive segmentation network, improves GTA5→Cityscapes and Synthia→Cityscapes mean intersection-over-union over the same pipeline without the filter. It also outperforms three weaker versions of the same idea: an unconstrained linear map trained with the same losses, a fixed principal-component truncation, and a closed-form "domain-contrastive" eigenbasis. Finally, the directions it discards should be highly predictive of domain identity and nearly useless for predicting class.
Concretely, encoder features $f \in \mathbb{R}^d$ at each decoder input level are replaced by $W W^\top f$. This keeps the dimension, so the decoder is unchanged, while deleting the $d-p$ directions orthogonal to the columns of $W$. The filter is trained with
$$\min_{W \in \mathrm{St}(p,d)} \; \mathcal{L}_{\text{seg}}^{\text{src}} + \mathcal{L}_{\text{self}}^{\text{tgt}} + \lambda\, \mathrm{MMD}^2\!\left(W^\top f_{\text{sim}},\, W^\top f_{\text{real}}\right).$$
The terms are as follows. $\mathcal{L}_{\text{seg}}^{\text{src}}$ is the supervised loss on synthetic images. $\mathcal{L}_{\text{self}}^{\text{tgt}}$ is the pipeline's existing pseudo-label loss on real images. $\mathrm{MMD}^2$ is a kernel maximum mean discrepancy, a non-adversarial distance between the two feature distributions. $\lambda$ weights the domain term. Updates use the Riemannian gradient followed by a polar or Cayley retraction, which is a step that maps the update back onto the manifold. Because the constraint forbids rescaling, the only way to reduce the discrepancy term is to change which directions are kept.
Relation to Prior Work
Two lines of work motivate the subspace view of domain shift. The Geodesic flow kernel represents source and target as principal subspaces on the Grassmannian, the space of linear subspaces, and integrates along the geodesic between them. Its descendants, Grassmannian Graph-attentional Landmark Selection and kernel regression on the Grassmannian for zero-shot adaptation, add sample reweighting and metadata-driven subspace prediction. All three compute subspaces once, on frozen features, by variance-based principal component analysis (PCA), and test on Office/Caltech-scale classification. Here the subspace is learned with gradients through a dense-prediction network and chosen by domain-specificity rather than variance.
The most direct motivation is Driving with DINO. It finds that simulator "texture baking" lives in the minor principal components of foundation-model features and drops those components with static PCA to condition a video generator. This proposal takes that observation and asks whether it holds for downstream perception. It also asks whether "minor variance" is the right selection criterion at all, or whether a learned, domain-aware choice of directions does better. HRDA attacks the same "texture trap" differently, by mixing low-resolution context crops. The filter here works in feature space at full resolution, so it should not hurt thin classes the way downsampling can.
The method itself comes from Stiefel-constrained adaptation. StelLA keeps low-rank adapter bases orthonormal through polar retraction at about 15% overhead over LoRA, but it never studies domain shift. Calibrated Adaptation shows that flat-space inference projected onto the manifold inflates variance under shift. That result constrains the design: optimization will be intrinsic, not "step, then orthogonalize." A Primer on SO(3) Action Representations reports that SVD projection inside the network disrupts gradient flow. This is the reason to prefer a retraction applied to the parameters over a projection in the forward pass.
The proposal does not claim that Stiefel adapters, Grassmannian domain adaptation, or PCA feature truncation are new. What is new is a learned, orthonormal, domain-aware deletion of feature directions inside a modern synthetic-to-real segmentation pipeline, together with an analysis of what gets deleted.
Research Program
The host pipeline is the DAFormer codebase (lhoyer/DAFormer) with its hierarchical transformer encoder, extended with HRDA multi-resolution training and MIC masked consistency. MIC is the strongest reported method in the file. One filter is inserted per encoder level feeding the decoder. Each filter is initialized to the top-$p$ principal subspace of pooled source features, so training starts exactly at the PCA baseline and any gain is attributable to learning. The kept fraction $p/d$ is swept from 0.5 to 0.95. Retractions on matrices of at most 512 columns add negligible cost relative to the encoder.
The program has three stages. First comes a mechanism study on VisDA-2017, where synthetic-to-real classification is cheap and linear probes are easy to interpret. Second is the headline test on GTA5→Cityscapes and Synthia→Cityscapes. Third, if the effect is real, is a transfer of the claim to sensor shortcuts on SynLiDAR→SemKITTI, where the shortcut is missing-point patterns rather than texture.
Throughout, the discarded subspace $I - WW^\top$ is analyzed. A linear probe measures domain accuracy and class accuracy on the kept and discarded features. The discarded directions are compared across random seeds by principal angles, to check whether there is a stable "simulator subspace" or whether each run finds something arbitrary.
Baselines
The naive reference point is the source-only model trained on GTA5 and evaluated on Cityscapes with no adaptation. It shows how much of the gap any method closes. The file gives no source-only number for the transformer encoder on GTA5→Cityscapes, so it will be measured.
The strongest existing method is MIC, which reports 75.9 mIoU on GTA5→Cityscapes and 92.8% accuracy on VisDA-2017. Below it, HRDA reports 73.8 on GTA5→Cityscapes and 65.8 on Synthia→Cityscapes, and DAFormer reports 68.3 on GTA5→Cityscapes. No reported result was found on Lacuna for MIC on Synthia→Cityscapes, so that number will be reproduced. MIC will be rerun in our hands with its published schedule. The filter variant uses identical data, iterations, crops, and a matched hyperparameter search. The filter adds only $\lambda$ and $p$, and MIC receives an equal number of extra tuning trials over its own masking ratio and loss weight.
Beating MIC means two things: a mean improvement over three seeds larger than twice the seed standard deviation, and no regression on the five thin or rare classes. The thin-class check uses per-class IoU from our reruns, since the file reports per-class numbers only for train and truck under DAFormer.
Four ablations isolate the key ingredient, each using exactly the filtered pipeline's losses and $p$. The unconstrained ablation replaces $W$ with a free $d\times p$ matrix, testing whether orthonormality matters. The soft-penalty ablation adds $\lambda'\|W^\top W - I\|_F^2$ instead of the hard constraint, testing whether a cheap approximation suffices. The static-PCA ablation freezes $W$ at the source principal subspace, as in Driving with DINO, testing whether learning matters. The project-after-step ablation takes a Euclidean step and then orthogonalizes by SVD, testing whether intrinsic optimization matters, as the SO(3) and Calibrated Adaptation papers suggest. For LiDAR, the reference is Adversarial Spatial Masking, reported in Adversarially Masking Synthetic to Mimic Real at 22.8 mIoU on SynLiDAR→SemKITTI, against a 15.2 source-only baseline and a 55.5 oracle.
Why not just…?
Why not just keep the top principal components, as Driving with DINO does? This is cheap and has evidence behind it in a generative setting. But variance and domain-specificity are different properties. A high-variance direction can be pure simulator lighting, and a low-variance direction can carry thin-object boundaries. The static-PCA ablation settles this directly. Because the learned filter starts from that PCA solution, any learned gain over it is attributable to learning rather than initialization. If PCA matches the learned filter, the proposal's contribution reduces to "Driving with DINO's trick transfers to segmentation," and we will report it that way.
Why not just solve for the subspace in closed form? If both the task signal and the domain signal were quadratic, for example between-class scatter versus between-domain scatter, the best orthonormal $W$ would be a generalized eigenbasis, with no Riemannian optimization needed. This is the most serious threat, and it is included as an explicit baseline. Its eigenbasis is recomputed every few thousand iterations as the encoder changes. The argument for the iterative version is that the real objective runs through a nonlinear decoder and moving self-training targets, so the quadratic surrogate may pick the wrong directions. If the recomputed eigenbasis matches the learned filter on GTA5→Cityscapes, we will recommend it as the simpler method.
Why not just use an unconstrained linear map or a soft orthogonality penalty with the same domain loss? This is the alternative the orthonormality argument predicts will fail. A free map can lower the discrepancy term by shrinking all features, damaging semantics. Evidence that unconstrained low-rank factors drift and underperform comes from StelLA, but only outside domain adaptation. The unconstrained and soft-penalty ablations test it here. The prediction is a measurable collapse: the singular values of the free map shrink, and the class-probe accuracy on its outputs drops.
Why not just rely on MIC's masking and self-training, which already push the model off local texture? This may be enough, and if so the filter adds nothing. MIC targets context use on real images. It contains no explicit term that removes feature directions encoding "this came from a renderer." The headline comparison is filtered MIC against MIC under matched tuning. A null result here is informative: it would say that the self-training pipeline already removes simulator directions implicitly. We will check this by running the domain probe on MIC's own features.
Proposed Experiments and Decisive Checks
On VisDA-2017, we first measure how much domain identity is linearly decodable from encoder features, and how this varies across principal components ordered by variance. This tests whether simulator information concentrates in a few directions at all. If domain identity is spread nearly uniformly, the premise fails early and cheaply.
We then train the filter on VisDA and on GTA5→Cityscapes and Synthia→Cityscapes against MIC, HRDA, DAFormer, source-only, and the four ablations plus the closed-form eigenbasis. Each configuration uses three seeds, with mIoU and per-class IoU as metrics.
Three outcomes are decisive. The first is whether the filtered pipeline beats MIC by more than seed noise while also beating the static-PCA and closed-form baselines. The second is whether, on held-out images, the discarded subspace gives domain-probe accuracy near 100% and class-probe accuracy near chance, while the kept subspace shows the reverse. The third is whether the discarded subspaces from different seeds have small principal angles, indicating a reproducible simulator subspace rather than arbitrary deletions.
A fourth check is cross-simulator. A filter learned on GTA5→Cityscapes is frozen and applied to Synthia→Cityscapes. Partial transfer would suggest the deleted directions encode something generic about rendered images rather than one game's artifacts. Only if these checks pass do we port the filter to the point-cloud network on SynLiDAR→SemKITTI. There the prediction is that the deleted directions align with the beam-dropout statistics that Adversarial Spatial Masking imitates by masking input points.
References
- Geodesic flow kernel for unsupervised domain adaptation
- Grassmannian graph-attentional landmark selection for domain adaptation
- Zero-Shot Domain Adaptation via Kernel Regression on the Grassmannian
- Driving with DINO: Vision Foundation Features as a Unified Bridge for Sim-to-Real Generation in Autonomous Driving
- HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation
- StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
- Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuning
- A Primer on SO(3) Action Representations in Deep Reinforcement Learning
- MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation
- DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation