Networks trained on rendered images often latch onto cues that exist only in the renderer: clean edges, uniform lighting, ray-tracing patterns, fixed viewpoints. These cues are easy to learn, so the network learns them first, and they vanish on real photographs. We propose to find these cues directly and cheaply, then forbid the network from learning along them.
The tool is the empirical neural tangent kernel, a similarity between two images measured by how alike the network's parameter gradients are on them. In this kernel's spectrum we look for directions that carry a lot of energy on synthetic images and little on unlabeled real images. Computing the full kernel on a few hundred thousand images is impossible, so we approximate it with the Nyström method. That method builds a low-rank stand-in for the kernel from a few thousand sampled "landmark" points. During training, gradient updates are projected away from the simulator-only directions we found. The claim is that, at the scale of VisDA's 152k rendered images, a scalable spectral analysis can name and remove simulator shortcuts. This is a different operation from matching the two domains' statistics, which is what kernel-based domain adaptation has done so far.
Core Claim and Where It Comes From
Three prior results set up the proposal. The first motivates it. Shortcut Features as Top Eigenfunctions of NTK proves that shortcut features align with the top eigenfunctions of the tangent kernel. These features have large eigenvalues because they are highly "available," and that availability lets them dominate early gradient descent regardless of any max-margin bias. That paper stops at diagnosis and small debiasing experiments on Waterbirds and CelebA.
The second supplies the method. KREPES shows that a Nyström approximation of the empirical tangent kernel scales to ImageNet-1K on one GPU in $O(n\sqrt{n})$ time. Its kernel surrogate stays within 1% accuracy of the original network in most cases and has Kendall-τ above 0.8 on decision rankings. KREPES uses this only to audit self-supervised models, never across domains.
The third constrains the design. Kernel methods for domain transfer have so far meant aligning distributions: DAN's mini-batch Maximum Mean Discrepancy (MMD) in VisDA. MMD is a kernel distance between two sets of samples. Rethink MMD proves that minimizing it inflates within-class variance and blurs class boundaries.
Our proposal therefore does not align anything. It removes a subspace.
A plain "remove the top eigenfunctions" rule is wrong for this problem, because class identity also lives in the top of the spectrum. The new ingredient is a cross-domain contrast. Let $K_S$ and $K_T$ be the Nyström tangent-kernel matrices restricted to source (synthetic) and target (real, unlabeled) landmarks, expressed in a shared landmark basis. We solve the generalized eigenproblem
$$K_S v = \rho\,(K_T + \epsilon I)\,v,$$
where $\rho$ is the ratio of a direction's energy on synthetic data to its energy on real data and $\epsilon$ is a small regularizer. Directions with large $\rho$ are what the network can easily read on synthetic images but finds almost absent on real ones. That is a working definition of a simulator shortcut. Directions shared by both domains, including class-discriminative ones, have $\rho$ near one and are left alone.
Research Program
Every $T$ steps we refresh the analysis. We sample $m$ landmarks, split between the two domains, and compute per-example gradients of the last few network blocks, randomly projected to a manageable width. From these we form the Nyström factors and solve the eigenproblem above at $O(m^3)$ cost. Because the tangent kernel equals $JJ^\top$ for the gradient matrix $J$, each eigenvector $v$ maps to a parameter-space direction $J^\top v$. Between refreshes, the supervised loss gradient on synthetic labels is projected onto the orthogonal complement of the top-$k$ high-$\rho$ directions. In the linearized picture, this stops the function from changing along simulator-only eigenfunctions. Labels are needed only on synthetic data.
Choosing $k$ and $\epsilon$ without target labels is the main practical risk. We will choose $k$ from a gap in the sorted $\rho$ spectrum and report sensitivity across the full range.
The analysis side asks three questions. First, does the method find the cues it should? We test this on a controlled shortcut injection described below. Second, how does the identified subspace change as training proceeds? Third, how much does Nyström rank matter? Here we reuse the effective-dimension argument of Lin et al. as a heuristic for how many landmarks are enough. We also expect data-dependent landmarks to beat random Fourier features, following the $O(1/m)$ versus $O(1/\sqrt{m})$ bounds of Yang et al..
Baselines
The naive reference is source-only training on synthetic data with the same backbone. VisDA reports that an AlexNet trained this way drops "from 99% accuracy on synthetic data to just 28% on real COCO images." We will rerun source-only with our backbone rather than quote that figure, because the backbones differ.
The kernel-alignment reference is DAN, which reaches 51.62% on VisDA-C in VisDA. We will reimplement it on our backbone with an equal tuning budget.
The strongest existing method is self-ensembling (mean teacher with confidence thresholding), which won VisDA-2017 with 92.8% test accuracy in Self-ensembling for visual domain adaptation. We will run it with its published augmentation recipe, the same backbone, the same number of epochs, and the same hyperparameter search budget as our method. We will also run self-ensembling combined with our projection.
Beating self-ensembling alone is not the bar, since it relies on hand-designed augmentations. The bar is either that the combination improves over self-ensembling by more than seed variance, or that our method alone matches it in a setting where its augmentations are disabled or mismatched.
The spectral debiasing reference is Spectral Decoupling, as evaluated in the NeurIPS 2025 shortcut paper. It penalizes logit magnitude and uses no target data.
Three ablations isolate the key ingredient. The first removes the top-$k$ eigenfunctions of the source kernel alone, with no cross-domain ratio. This tests whether the contrast with real data matters. The second solves the same ratio eigenproblem on penultimate-layer features instead of tangent-kernel gradients. This tests whether the tangent kernel matters at all. The third projects out $k$ random directions, which controls for capacity loss. A further check compares Nyström against exact tangent-kernel computation on a 20k subsample, measuring subspace overlap and accuracy, to confirm the approximation is not doing the work by accident.
Why not just…?
Why not just render more synthetic data? Johnson-Roberson et al. show that scaling from Sim 10k to Sim 200k lifts Easy KITTI to 0.6803 mAP. That beats real Cityscapes at 0.6247, so scale helps. However, VisDA's 152k renderings still give 28% on COCO without adaptation. More renderings from the same engine reproduce the same artifacts. Our injection experiment tests this directly by varying the source set size while holding the artifact fixed.
Why not just use self-ensembling? This is the real threat, since it holds 92.8% on VisDA-C. Its gains depend on domain-specific augmentations such as color inversion, and its class-balance loss assumes a uniform target class distribution, as French et al. note. If our projection adds nothing on top of self-ensembling and also fails to help when augmentations are removed or the target classes are imbalanced, we will conclude that the idea is not useful for vision. The proposed experiments will settle this either way.
Why not just align the domains with MMD or a closed-form Gaussian map? Alignment pulls the two distributions together. Rethink MMD proves that this blurs classes, and DAN's 51.62% reflects the cost. Affine Transport is fast but linear. It reduced hockey-puck prediction error from 0.73 to 0.08 in a low-dimensional setting, but it cannot reach non-linear cues. Our method leaves shared structure alone and acts only on directions that real data lacks. The DAN comparison and the feature-space ablation separate these effects.
Why not just do this in feature space with PCA? Penultimate features are cheaper to compute. But they describe what the network already represents, not what it will learn next. The tangent kernel governs how easily the training dynamics pick up a cue, which is where the shortcut theorem operates. The feature-space ablation is decisive here: if it matches our method, the tangent kernel is unnecessary and we will say so.
Proposed Experiments
The fast sandbox is MNIST→MNIST-M, where PixelDA reports 98.2% (PixelDA). There we build a controlled test by adding a renderer-style artifact to the source images: a faint, class-correlated texture stamp that never appears in the target. The decisive check is whether the high-$\rho$ subspace captures the stamp. We measure this by the correlation between the stamp indicator and the top high-$\rho$ eigenfunctions, and by target accuracy recovering to the level of training without the stamp. The pure-spectral ablation should also remove class directions and lose accuracy.
The main benchmark is VisDA-C (152k CAD renderings to 55k COCO validation images, with the YouTube-BB test set). We report mean per-class accuracy over three seeds, along with wall-clock time and memory against the landmark count $m$. VisDA itself warns that ImageNet pretraining confounds sim-to-real conclusions, so we run both ImageNet-initialized backbones and backbones pretrained self-supervised on synthetic images only. We expect the effect to be larger in the second case.
The robustness check repeats VisDA-C with a subsampled, class-imbalanced target set. This is where self-ensembling's uniform-class assumption should hurt and our label-free spectral criterion should not, unless imbalance pushes class directions to high $\rho$. We will report the $\rho$ spectrum to check for that.
The idea is supported if three things hold together: the injected artifact is recovered, the cross-domain ratio beats both pure-spectral and feature-space ablations, and either the combination with self-ensembling or the imbalanced setting shows a gain beyond seed variance.
References
- Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More
- Interpretable Self-Supervised Learning via Representer Landmarks and Nyström Approximation
- VisDA: The Visual Domain Adaptation Challenge
- Rethink Maximum Mean Discrepancy for Domain Adaptation
- Optimal Rates of Sketched-regularized Algorithms for Least-Squares Regression over Hilbert Spaces
- Nyström Method vs Random Fourier Features: A Theoretical and Empirical Comparison
- Self-ensembling for visual domain adaptation
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Affine Transport for Sim-to-Real Domain Adaptation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks