Networks trained on simulator data often fail on real data. The usual explanation is that they latch onto "shortcuts": cues such as rendering signatures, textures tied to particular CAD models, or the absence of real sensor dropout, which predict the label in simulation but not in reality. Theory has a precise version of this story. In a linear model on Gaussian data split into "core" and "spurious" features, whether robustness interventions help or hurt depends on which feature group separates the classes more strongly. On the Effects of Adversarial Perturbations on Distribution Robustness shows that adversarial perturbations improve out-of-distribution robustness when core features are more separable than spurious ones, and harm it otherwise. The catch, which this literature names as an open problem, is that nobody knows how to measure those two separabilities in a real, non-linear network.
We propose that sim-to-real transfer is the setting where this quantity becomes measurable. Unsupervised adaptation always gives us labeled synthetic data and unlabeled real data. The directions in a layer's feature space that tell simulation apart from reality are, by definition, the directions that will shift at deployment. How much class information a network routes through those directions, compared with how much it routes through the rest of the space, is a label-free, layer-wise stand-in for the theory's spurious-versus-core separability. Our core claim has two parts. First, this ratio, computed before any real labels are seen, predicts real-domain accuracy better than simulator validation accuracy or a sim-versus-real discriminator. Second, it predicts the sign of the effect of adversarial training on transfer, as the linear theory says it should. The note then tests whether penalizing the ratio during end-to-end training beats the existing subspace methods.
Core Claim
Let $h_\ell(x)$ be the features at layer $\ell$. We fit a linear discriminant between synthetic and real features and take its top-$k$ directions as the domain subspace $D_\ell$, with $D_\ell^\perp$ its complement. For any subspace $U$, let $m(U)$ be the average Mahalanobis distance between synthetic class means after projecting onto $U$, computed with the within-class covariance in $U$. This is the separability measure of On the Effects of Adversarial Perturbations on Distribution Robustness, applied to learned features instead of raw coordinates. The score is
$$S_\ell = \frac{m(D_\ell)}{m(D_\ell^\perp)},$$
the ratio of class separability carried by domain-varying directions to class separability carried by domain-stable directions. We expect three things. Real-domain error should rise with $S_\ell$ at late layers. Adversarial training should help transfer when $S_\ell < 1$ and hurt when $S_\ell > 1$, which is the non-linear, feature-space analogue of the condition $m_1 > m_2$. And a training penalty that lowers $m(D_\ell)$ while leaving domain information that carries no class signal untouched should transfer better than methods that remove the whole domain subspace.
One distinction matters for the theory. Accuracy on the wrong line: On the pitfalls of noisy data for OOD generalisation blames out-of-distribution breakdown on memorizing label noise through nuisance features. Simulator labels are essentially noise-free, though, so a sim-to-real shortcut must be driven by separability, not by noise memorization. This is why we build on separability rather than on the benign-overfitting results.
Relation to Prior Work
Three recent papers already use linear structure to fight synthetic-data shortcuts, and they motivate this proposal.
Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning (the DREAM method) finds the dominant synthetic-versus-real directions by singular value decomposition of extracted features and suppresses them after training. We differ in three ways: we measure class information inside that subspace rather than deleting the subspace, we act during training, and we target standard sim-to-real rather than incremental replay.
Domain Generalization with Nuclear Norm Regularization encourages low-rank features with a label-free penalty and proves out-of-domain guarantees only for linear models. It never uses the unlabeled target data that adaptation provides.
Compositional by Design: Background-Invariant Representations via Linear Additivity in VLMs averages away background vectors in a frozen vision-language model. It constrains our design, because it reports that the approach fails for nuisances intrinsic to the object, such as texture or material, and that fine-tuning brings the shortcut back. Our score must therefore work on unfrozen networks and on intrinsic nuisances, and we test exactly that.
On how to avoid exacerbating spurious correlations when models are overparameterized supplies a competing loss, vector-scaling loss, which reshapes margins and is proven only for linear classifiers.
On the theory side, Benign Overfitting without Linearity supplies the analysis technique: tracking gradient descent in two-layer leaky-ReLU networks far from initialization.
Research Program
The work has three parts.
Theory. We will attempt a modest result. Take a two-layer smoothed leaky-ReLU network trained by gradient descent on a mixture with core coordinates and class-conditional "simulator artifact" coordinates that vanish at test time. The target is to show that the learned first-layer weight mass on artifact directions, relative to core directions, grows with the input-space separability ratio. That would justify using $S_\ell$ at the hidden layer. We expect to borrow the loss-ratio control of Benign Overfitting without Linearity. We treat this as a target, not a promise: if it holds only for $p \gg n$, we will say so.
Diagnostic study. We will train a large population of models and test whether $S_\ell$, computed with no real labels, predicts real accuracy. The population covers ResNet and ViT backbones, augmentation strengths, adversarial-training radii, synthetic dataset sizes, and with or without pixel translation.
Intervention. We add $\lambda\, m(D_\ell)$ as a differentiable penalty. $D_\ell$ is re-estimated every few hundred steps from a buffer of synthetic and unlabeled real features. Lowering $m(D_\ell)$ pushes class information out of domain-varying directions without forcing the domains to look identical.
Baselines
The naive reference point is source-only training. On VisDA-C, AlexNet reaches 99% on synthetic data and 28% on real COCO images (VisDA). On SynLiDAR to SemanticKITTI, source-only scores 15.2% mIoU (mean intersection-over-union, the standard segmentation overlap metric) (Adversarially Masking Synthetic to Mimic Real). We rerun source-only on our own backbones so that every comparison shares an architecture.
For prediction, there are two naive predictors. The first is synthetic validation accuracy, which is the "accuracy on the line" assumption. The second is the accuracy of a sim-versus-real classifier on the same features, which measures how different the domains look without asking whether that difference carries class signal. $S_\ell$ beats them if its Spearman rank correlation with real accuracy is higher by a margin outside bootstrap intervals across the model population.
The strongest existing methods are domain-specific. On VisDA-C, the reported challenge winner reached 92.8% using pseudo-labeling and a mean teacher (VisDA). On SynLiDAR to SemanticKITTI, adversarial masking reaches 22.8% mIoU against an oracle trained on real data at 55.5% (Adversarially Masking Synthetic to Mimic Real).
We also run DREAM's post-hoc projection, the nuclear-norm method, and vector-scaling loss on the same backbones. Each gets the same data, the same epochs, and an equal tuning budget of 20 trials per method selected on synthetic validation; a second selection track uses $S_\ell$ for every method. There is no reported result found on Lacuna for DREAM, the nuclear-norm method, or vector-scaling loss on VisDA-C or SynLiDAR, so these are new measurements.
Beating them means a statistically reliable gain over the best of these on real-domain accuracy or mIoU at matched compute. Because our penalty is orthogonal to pseudo-labeling, we additionally test whether stacking it on mean-teacher pseudo-labeling raises the reproduced winner's score.
Two ablations isolate the key ingredient. The first penalizes class separability in a random subspace of the same dimension $k$, which tests whether the domain directions matter. The second removes all of $D_\ell$ during training, an in-training DREAM, which tests whether keeping class-free domain information matters.
Why not just…?
Why not just scale up and diversify the simulator? Driving in the Matrix is real support for this. Going from 10k to 200k GTA V frames reached 0.6803 mAP on KITTI Easy, above the 0.6247 of a model trained on real Cityscapes. Yet SynLiDAR's billions of points still leave source-only at 15.2% mIoU. Our study includes synthetic dataset size as a factor. If scaling lowers $S_\ell$ and improves transfer in lockstep, that is consistent with our account, because it says scaling works only when it dilutes class information in domain-varying directions. If scaling improves transfer while $S_\ell$ stays flat, our account is wrong.
Why not just translate synthetic images to look real? PixelDA reached 98.2% on MNIST to MNIST-M (Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks). Its own authors limit it to low-level appearance, and DREAM's ablation shows raw generated replay collapsing to 33.50% because of artifacts introduced by the generator. We run PixelDA-translated data as one population member and as a baseline. Translation is not a rival to the score but a case the score should explain: a good translation should lower $S_\ell$.
Why not just freeze a foundation model and train a linear probe? This is a real threat on VisDA-C. We include a frozen vision-language model with a linear probe and with the background-averaging method. The reported failures of the background-averaging method on intrinsic nuisances and under fine-tuning (Compositional by Design) suggest it will lose on LiDAR and on texture-driven classes. If the frozen probe wins everywhere, we will report that the penalty is unnecessary when strong frozen features exist, and restrict the claim to domains such as LiDAR without such backbones.
Why not just use adversarial training or domain-adversarial alignment? On the Effects of Adversarial Perturbations on Distribution Robustness predicts that adversarial training helps or hurts depending on separability, so its sign is our test, not our competitor. Domain-adversarial alignment forces the domains to look identical, and adversarial masking found that such discriminators chase the trivial missing-point cue. The in-training full-subspace ablation measures what that costs.
Proposed Experiments
We use four benchmarks. MNIST to MNIST-M and Waterbirds serve as controlled cases where the nuisance is known. VisDA-C covers CAD-to-real classification, including intrinsic texture nuisances. SynLiDAR to SemanticKITTI and nuScenes covers sensor-structured shifts.
The first decisive check is predictive. Across about 200 trained models per benchmark, $S_\ell$ at the penultimate layer must outrank both naive predictors.
The second is a sign test. For each base model, we predict from $S_\ell$ whether adversarial training will raise or lower real accuracy, then train it and score the agreement. Chance is 50%, so agreement well above that across the model population is evidence that the linear condition carries over to non-linear features.
The third is the intervention comparison under the matched protocol above, together with both ablations.
The proposal fails cleanly if any of three things happens: the score's rank correlation is no better than the domain-discriminator baseline, sign agreement is near chance, or the random-subspace ablation matches the full penalty.
References
- On the Effects of Adversarial Perturbations on Distribution Robustness
- Accuracy on the wrong line: On the pitfalls of noisy data for OOD generalisation
- Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
- Domain Generalization via Nuclear Norm Regularization
- Compositional by Design: Background-Invariant Representations via Linear Additivity in VLMs
- On how to avoid exacerbating spurious correlations when models are overparameterized
- Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data
- VisDA: The Visual Domain Adaptation Challenge
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks