A simulator knows how every training image was made: the light direction, the camera angle and distance, the texture it applied. Real photographs come with none of this. Most sim-to-real methods throw that free information away. They try instead to make synthetic and real feature distributions look alike, or they corrupt synthetic inputs until a domain classifier is fooled. We propose to use it directly. We treat the logged rendering parameters as nuisance variables, find the part of a frozen network's features that depends on them, including non-linear dependence, and remove that part in closed form before training the classifier. The removal step is a fixed projection learned on synthetic data only, so it applies unchanged to real images.
The mechanism comes from algorithmic fairness. Extending Fair Null-Space Projections for Continuous Attributes to Kernel Methods introduced Fair Kernel Decomposition (FKD). FKD removes all non-linear correlation between learned features and continuous protected attributes such as age or income by projecting the kernel feature space onto the subspace those attributes cannot predict. It was developed and tested only on tabular regression. Our claim is that the sim-to-real setting is a better fit for this machinery than fairness is. The nuisance attributes are exactly known and continuous, they are plentiful, and the "shortcut" a network learns in simulation is plausibly carried by features that track rendering conditions. The research question is concrete. On 12-class synthetic-to-real classification, does removing the non-linear dependence of frozen features on logged rendering parameters close a measurable share of the gap? And does it do so beyond what linear removal, removal of random directions, or simply balancing the rendering parameters achieves?
Core Claim
The claim has two parts. First, a substantial fraction of what separates synthetic from real features in a pretrained backbone can be predicted, non-linearly, from the simulator's rendering parameters. Second, projecting that predictable component out of the features improves accuracy on real target images. This holds without target labels and without changing the backbone.
The approach differs from three groups of prior work.
Global distribution matching, exemplified by the Deep Adaptation Network evaluated in VisDA: The Visual Domain Adaptation Challenge, aligns whole feature distributions using Maximum Mean Discrepancy, a distance between distributions computed in a kernel feature space. It does not know which feature directions are nuisance, so it can align simulator artifacts with real noise rather than removing them.
Input corruption, exemplified by Adversarially Masking Synthetic to Mimic Real, drops synthetic LiDAR points adversarially so that a domain discriminator cannot use sensor-density cues. That approach works on inputs and needs a hand-designed corruption family. Ours works on representations and needs only the logs the simulator already produces.
Kernel manifold alignment, in Kernel Manifold Alignment for Domain Adaptation, also solves a closed-form kernel problem, but it needs labeled target examples to build its cross-domain graphs. It was validated on small, hand-crafted-feature benchmarks. Our method uses no target labels.
Method
Let $f(x)$ be the features of a frozen pretrained backbone for image $x$. Let $z(x) \in \mathbb{R}^m$ be a Nyström approximation of a Gaussian-kernel feature map of $f(x)$. Nyström is a low-rank approximation built from $m$ landmark samples, and FKD reports it is qualitatively identical to the exact computation when using only 25–50% of components. Let $r \in \mathbb{R}^p$ be the logged rendering parameters of a synthetic image, and $\psi(r) \in \mathbb{R}^q$ a kernel feature map of those parameters, so that non-linear functions of lighting or viewpoint are also covered.
On synthetic data we compute the cross-covariance $C = \mathrm{Cov}(z, \psi(r)) \in \mathbb{R}^{m \times q}$ and take $W$, its top-$k$ left singular vectors. These are the directions in feature space most predictable from rendering conditions. The scrubbed features are
$$ \tilde z(x) = \left(I - W W^\top\right) z(x), $$
and, following FKD's iterative scheme, we repeat this until a held-out kernel ridge regressor can no longer predict $\psi(r)$ from $\tilde z$ above chance. A linear or kernel ridge classifier is then trained on $\tilde z$ for synthetic images and applied to $\tilde z$ for real images. The projection is a fixed $m \times m$ matrix, so real images need no rendering parameters at test time. The total cost is one pass of feature extraction plus decompositions of $m \times m$ and $m \times q$ matrices. That is minutes on a single GPU with $m$ in the low thousands, which avoids the $O(n^2)$ Gram-matrix cost that limits exact kernel methods.
There is an obvious failure mode. Some rendering parameters may genuinely carry class information, because different object classes may be rendered from systematically different viewpoints. Scrubbing those parameters could remove signal along with nuisance. We handle this explicitly by also running a class-conditional variant that removes only the within-class dependence on $r$, computing $C$ from class-centered features. Comparing the two variants tells us which effect dominates.
Research Program
The first stage is diagnostic and can fail cleanly. We build a synthetic source set with the same 12 classes as VisDA-C, rendering CAD models with logged light direction and intensity, camera azimuth, elevation and distance, background type, and texture identity. We do this because we cannot confirm from our sources that the original VisDA-C renders ship with this metadata. We then measure how well a kernel domain classifier separates these renders from real VisDA-C images using frozen features. Next we measure how much that separability drops after scrubbing $r$. If scrubbing barely reduces separability, rendering parameters do not explain the gap, and the main claim is falsified before any accuracy comparison.
The second stage measures accuracy on real images. The third stage tests whether the method is complementary to target-side methods: we feed scrubbed features into a pseudo-labeling or clustering stage of the kind used by Make the U in UDA Matter.
A built-in negative control uses LineMod pose estimation from Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks. There, viewpoint is the target variable, so scrubbing it must collapse pose accuracy. If it does not, the projection is not removing what we claim it removes.
Baselines
The naive reference point is source-only training: the same frozen backbone and the same classifier head on unscrubbed synthetic features, evaluated on real images. VisDA: The Visual Domain Adaptation Challenge reports that an AlexNet source-only model falls from 99% synthetic accuracy to 28% on real COCO images. We rerun source-only with our backbone rather than reuse that figure, because the backbones differ.
The closest kernel-based competitor is the Deep Adaptation Network. That paper reports DAN at 51.62% accuracy on VisDA-C. We reimplement it as multi-kernel MMD between synthetic and real features, applied on top of the same frozen backbone through a small trainable adapter. The adapter gets the same compute and the same hyperparameter-search budget as our method, which tunes kernel bandwidth, $m$, $k$ and the number of iterations. Our method beats DAN if it matches or exceeds DAN's real-image mean-class accuracy while using no target images during fitting.
The strongest existing methods are target-side. ICON reports 87.4% mean-class accuracy on VisDA-2017 in Make the U in UDA Matter, and the VisDA-C challenge winner reached 92.8% through pseudo-labeling. Both fine-tune backbones end to end, so a frozen-feature projection alone is not expected to beat them. The fair test is complementarity. We run ICON-style clustering and pseudo-labeling with the same backbone and schedule, once on raw features and once on scrubbed features. Scrubbing beats this baseline if it adds accuracy on top of it.
We also compare against test-time entropy minimization with Tent, which updates batch-norm scale and shift on real batches. Tent reports raising GTA5→Cityscapes segmentation IoU from 28.8% to 35.8%. We found no reported Tent result on VisDA-C classification in our sources, so we run it ourselves.
Four ablations isolate the key ingredient:
- Linear scrubbing on raw $f(x)$ with linear $r$. This is iterative null-space projection without kernels and tests whether non-linearity matters.
- Random scrubbing, which removes $k$ random directions of the same rank. This tests whether gains are merely a regularization effect.
- Domain-label scrubbing, which projects out directions predictive of the binary synthetic-versus-real label using unlabeled real images. This tests whether the rendering logs add anything beyond domain identity.
- Class-conditional versus marginal scrubbing, which addresses the class-correlated-viewpoint failure mode described above.
Why not just…?
Why not just randomize the rendering parameters more widely? This is the most serious threat. If lighting and viewpoint vary independently of class, a classifier might already ignore them. But varying a nuisance does not force the features to stop encoding it. Pure data scaling in Driving in the Matrix beat real-data training on KITTI "Easy", with 0.6803 mAP versus 0.6247, yet fell to 0.4207 on "Hard", so scale alone leaves a gap. We test this threat directly. We train a source-only model on a resampled set in which every rendering parameter is balanced across classes, and we also report the scrubbed result on top of that balanced set. If balancing alone matches scrubbing, we will report that the mechanism is unnecessary.
Why not just align covariances or scrub linearly? Linear methods are cheaper and need no bandwidth tuning. The available evidence favors non-linear treatment: kernel manifold alignment reached 48.7% on Office-Caltech versus 37.4% for the linear Geodesic Flow Kernel in Kernel Manifold Alignment for Domain Adaptation, and FKD was motivated by linear null-space projection leaving non-linear correlations intact. That evidence comes from other tasks, however. The linear-scrubbing ablation settles the question here. If it recovers most of the gain, we will recommend the linear version.
Why not just scrub the synthetic-versus-real label, or use MMD, and forget the logs? Domain-label removal needs no simulator metadata and uses target data, which is a real advantage. Its weakness is that a single binary label defines very few directions to remove, and it can latch onto superficial cues. That discriminator-shortcut problem is what motivated Adversarially Masking Synthetic to Mimic Real. The domain-label ablation, run with an identical projection procedure, measures directly whether rendering logs remove more of the transferable gap.
Why not just fine-tune end to end with pseudo-labels? On final accuracy, ICON and pseudo-labeling likely win. We do not claim otherwise. The claim is that scrubbing is a cheap, label-free preprocessing step that should make such methods start from cleaner features, and the complementarity experiment tests exactly that.
Proposed Experiments and Decisive Checks
The primary benchmark is our logged-parameter 12-class synthetic source, evaluated on VisDA-C real validation images (COCO) and real test images (YouTube Bounding Boxes). We report mean-class accuracy. As a secondary check that the effect is not specific to object renders, we use GTA V frames with logged weather and time-of-day, captured through the pipeline of Driving in the Matrix. On these we evaluate a frozen-feature classifier on Cityscapes crops, with real-image accuracy as the metric. The LineMod negative control uses mean angle error as its metric.
Four results would decide the question:
- Kernel scrubbing of rendering parameters reduces sim-versus-real discriminability substantially more than random scrubbing of the same rank.
- It improves real-image accuracy over source-only and over the balanced-randomization baseline.
- It beats linear scrubbing.
- It adds accuracy when composed with pseudo-labeling.
The LineMod control must show pose error rising sharply when viewpoint is scrubbed. If checks 1 and 2 both fail, we conclude that rendering parameters are not where the transferable gap lives, which is itself a useful negative result for sim-to-real research.
References
- Extending Fair Null-Space Projections for Continuous Attributes to Kernel Methods
- VisDA: The Visual Domain Adaptation Challenge
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Kernel Manifold Alignment for Domain Adaptation
- Make the U in UDA Matter: Invariant Consistency Learning for Unsupervised Domain Adaptation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?