Models trained on simulated data usually fail on real sensors. The standard repair is to train a second network, a "domain discriminator", that tries to tell simulated features from real ones, and to push the main network until the discriminator is fooled. This proposal argues that the repair is aimed at the wrong space. Deep features have many dimensions, and in many dimensions any two finite sets of points can almost always be separated by a straight line. The discriminator therefore always has an easy win: it can rely on directions that encode sensor quirks, such as how many LiDAR points are missing, rather than directions that encode what objects are. Our idea is to first find the small set of feature directions that actually carry the label information, and then to align the two domains only inside that set. All other directions are left free to differ between simulation and reality.
In two sentences: a rank-limited layer learns the few directions that are sufficient to predict segmentation labels, and the simulated and real feature distributions are matched only in those directions. Because the matching happens in low dimension, the discriminator can no longer win by exploiting sensor-noise directions, and it stops distorting the features the classifier needs. We test this on synthetic-to-real LiDAR segmentation, where the gap between adapted and fully supervised models is still large: 22.8% versus 55.5% mean intersection-over-union (mIoU) on SemanticKITTI (Adversarially Masking Synthetic to Mimic Real).
Core Claim
Two results that are rarely read together motivate the claim. General stochastic separation theorems with optimal bounds shows that once dimension is large compared with $\log N$, exponentially many points can be split apart by a simple Fisher linear discriminant. How to Overcome Curse-of-Dimensionality for Out-of-Distribution Detection? proves that the error in estimating the density gap between two distributions grows with feature dimension. It also shows that throwing away irrelevant dimensions restores the contrast between distances. Taken together, these imply that a full-dimensional domain discriminator is partly measuring geometry rather than semantic mismatch. Its gradient is dominated by directions that are irrelevant to the task.
Li et al. observed the same failure from the applied side. In LiDAR adaptation, their feature-level discriminator separated domains by missing points rather than semantics. Their fix acts on the input: they learn to drop synthetic points until the scans look real.
We propose a fix in feature space. Let $z \in \mathbb{R}^D$ be a per-point backbone feature. The segmentation head first applies a rank-$d$ map $B^\top z$, with $B \in \mathbb{R}^{D \times d}$ and $d \ll D$. Following Neural Networks Perform Sufficient Dimension Reduction, we write this as a factorized first layer trained jointly with the rest of the head. That paper proves such a layer recovers the subspace on which the label's conditional mean depends. Adversarial alignment is applied only to $B^\top z$. The training objective is
$$\min_{\theta, B, h}\; \mathcal{L}_{\text{seg}}\big(h(B^\top z_\theta(x_s)), y_s\big) \;-\; \lambda\, \mathcal{L}_{\text{disc}}\big(B^\top z_\theta(x_s),\, B^\top z_\theta(x_t)\big),$$
where $x_s, y_s$ are synthetic scans and labels, $x_t$ are unlabeled real scans, $h$ is the nonlinear classifier, $\theta$ are backbone parameters, and the discriminator is maximized as usual. The claim to be tested is specific. Alignment restricted to the learned sufficient subspace should beat full-space alignment, and it should beat alignment in a random or PCA subspace of the same rank. If a random projection of equal rank works as well, the "sufficiency" part of the idea is wrong and only dimensionality matters.
There is an important caveat. The subspace is estimated from synthetic labels, so it can absorb simulator shortcuts that happen to predict labels in simulation. We treat this as something to measure, not something to hide. For each direction of $B$, we compute how well a linear probe separates simulated from real features. A direction that is both label-sufficient and strongly domain-separable is a candidate shortcut. Alignment concentrates its pressure on exactly those directions.
Relation to Prior Work
The closest prior method is Unsupervised Visual Domain Adaptation Using Subspace Alignment. It extracts separate PCA bases for source and target and aligns them in closed form. Its subspaces are unsupervised, so they follow the largest-variance directions, and in simulated data those are often rendering or sensor artifacts. It also works on fixed features with a shallow classifier. We instead learn the subspace from labels, jointly with a deep backbone, and use one shared subspace in which the distributions are matched.
Hvclip goes the opposite way. It expands CLIP embeddings to about 2.1 million dimensions and uses near-orthogonality to resist forgetting, reporting 92.5% on VisDA-2017. That result appears to contradict us. We think the two goals differ: HVCLIP uses high dimension to keep representations from drifting, whereas we restrict dimension so that a discrepancy measure stays informative. HVCLIP has also not been tested on dense geometric prediction. We do not adopt its method.
The Ghosal et al. paper supplies the motivating theory and the masking intuition. Its subspaces serve to reject out-of-distribution inputs, not to transfer across domains, and they require clean labels in the test domain's classes.
Unsupervised Contrastive Domain Adaptation for Semantic Segmentation aligns class prototypes across GTA5 and Cityscapes (60.2% mIoU). It already aligns something class-structured. However, its contrastive distances are computed in the full feature space, which is exactly where our argument says distance contrast weakens.
ASM is the strongest LiDAR competitor, and the two methods are complementary in kind. ASM changes inputs, and we change where the alignment loss is applied.
Research Program
The work has three parts.
The first is a diagnostic study, run before any new method. We train source-only and full-space adversarial models on SynLiDAR and measure four things on SemanticKITTI and nuScenes features: the ratio of nearest- to farthest-neighbor distances, the accuracy of a linear domain probe, and both of these again after projecting onto subspaces of rank $d$ chosen by the sufficient-dimension layer, by PCA, and at random. The prediction from stochastic separation is that the linear domain probe stays near perfect in full space even after adversarial training. It should drop toward chance only in a low-rank sufficient subspace. The second diagnostic measures how much of the discriminator's gradient energy falls in the complement of $B$.
The second part is the method itself, built in an existing LiDAR adaptation codebase such as yuan-zm/DGT-ST, with xiaoaoran/polarmix available for augmentation. We first warm up the rank-$d$ head on synthetic labels. We then turn on subspace-restricted alignment. Finally, as the third stage, we optionally add standard pseudo-label self-training.
Choosing $d$ is an acknowledged open problem in the sufficient-dimension paper. We sweep $d$ and also test a label-free selection rule: choose the smallest $d$ whose synthetic validation mIoU is within one point of the full-rank head.
The third part is the shortcut audit described above, followed by a variant that aligns only the most domain-separable sufficient directions.
Baselines
The naive reference point is source-only training on SynLiDAR. Li et al. report it at 15.2% mIoU on SemanticKITTI and 11.5% on nuScenes. We reproduce it with our backbone, and any method must clearly exceed our reproduced number.
The strongest existing method is Adversarial Spatial Masking. Li et al. report 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes. In the same table, CoSMix and SqueezeSegV2 reach 20.0% on SemanticKITTI. If we cannot run ASM's original code, we reimplement it on our backbone with the same synthetic and real data, iterations, and hyperparameter search budget as our method. We report both the reproduced and published figures. Beating ASM means exceeding the reproduced ASM by more than run-to-run variation over three seeds on both targets. We also test whether our feature-side restriction adds to ASM: input masking plus subspace alignment should exceed either method alone.
The full-space adversarial baseline is identical to our method except that the discriminator sees the full $z$. This isolates whether restricting alignment helps at all.
The key ablation is equal-rank alignment through a random projection and through a PCA projection fitted on pooled features. A PCA projection fitted per domain and aligned in closed form gives the Fernando et al. variant. Matching rank, data, and tuning budget means any gain specific to the sufficient subspace must come from its label-derived directions. A further control aligns only the complement of $B$. It should not help, and if it does, our story about which directions matter is wrong.
The reference ceiling is the fully supervised real-data oracle, reported by Li et al. at 55.5% on SemanticKITTI.
As a second testbed, we use GTA5→Cityscapes. There the comparison point is the contrastive method's 60.2% mIoU, and we apply our restriction to its prototype losses.
Why not just…?
Why not just put a small linear bottleneck before the discriminator? This is the most serious threat, and it is precisely our random and PCA projection ablations. If an arbitrary equal-rank projection gives the same gain, the effect comes only from lower dimension. In that case we would report that, and drop the claim that the subspace should be label-sufficient. We expect a difference for two reasons. Random projections approximately preserve pairwise geometry, so the sensor-noise directions remain visible to the discriminator. PCA favors high-variance directions, which the sufficient-dimension survey (Recent Advances in Supervised Dimension Reduction) notes can discard low-variance task information. Neither argument is proof, and the experiment decides.
Why not just use ASM's input masking and stop? ASM attacks the same shortcut at the input. It still leaves a gap of more than 30 points to the oracle, and it addresses only point dropout, not other feature-level differences such as reflectance or geometry statistics. Our combination experiment tests whether feature-side restriction recovers anything input masking misses. If ASM plus our method equals ASM alone, the two are redundant and our method's value reduces to simplicity.
Why not just self-train on pseudo-labels with mixing, as CoSMix does? Self-training is strong and orthogonal to alignment. CoSMix reaches 20.0% in Li et al.'s comparison. We therefore report our method both with and without a shared self-training stage. A gain that disappears once self-training is added would indicate that our alignment only produces better initial pseudo-labels, and we would say so.
Why not expand dimension instead, as HVCLIP does? HVCLIP's success is on image-level classification with CLIP, and the file has no evidence for dense LiDAR prediction. We include a hypervector-style random-expansion head at matched compute as an exploratory comparison on SemanticKITTI. A positive result there would directly contradict our thesis.
Proposed Experiments and Decisive Checks
The main experiments are SynLiDAR→SemanticKITTI and SynLiDAR→nuScenes. We report mIoU and per-class IoU over three seeds, with the GTA5→Cityscapes transfer as a secondary check.
Three outcomes would decide the question. First, in the diagnostic study, the linear domain probe should stay near perfect in full space after full-space adversarial training but fall toward chance within the sufficient subspace. If that does not happen, the stochastic-separation explanation is wrong. Second, sufficient-subspace alignment must beat equal-rank random and PCA alignment on both LiDAR targets. Third, the shortcut audit should find label-sufficient directions with high domain separability. Removing or aligning those directions should improve the classes most affected by dropout, such as road surfaces and specular objects.
A negative result on the second check would end the method claim. It would still leave a clean measurement of how much of the difficulty in adversarial adaptation comes from feature dimension alone.
References
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- General stochastic separation theorems with optimal bounds
- How to Overcome Curse-of-Dimensionality for Out-of-Distribution Detection?
- Neural Networks Perform Sufficient Dimension Reduction
- Unsupervised Visual Domain Adaptation Using Subspace Alignment
- Hvclip: High-dimensional vector in clip for unsupervised domain adaptation
- Unsupervised Contrastive Domain Adaptation for Semantic Segmentation
- Recent Advances in Supervised Dimension Reduction: A Survey