Compressed Kernel Tests as Label-Free Referees for Sim-to-Real Transfer

Teams that train on synthetic data face a practical problem that gets less attention than it deserves. They must pick a checkpoint, a noise-injection setting, or a subset of simulator frames, but they have no labels in the real domain to validate the choice. We propose to make that decision with a statistical test. The test is a kernel two-sample discrepancy, a measure of how different two sets of samples are, computed by comparing all pairs of points through a similarity function. The idea is to compare the entire synthetic feature set against the entire unlabeled real feature set, and to prefer the candidate whose features are statistically closest. Such tests are normally too expensive at simulator scale, because they cost time quadratic in the number of samples. Recent compression methods now summarize millions of points into small coresets while provably keeping test power. Those methods have been checked on Gaussians, downsampled EMNIST and tabular regression, but never on sim-to-real problems.

The bet is specific. A neural domain classifier is the usual cheap way to measure a domain gap, and it saturates on simulator data: it separates "clean synthetic" from "noisy real" almost perfectly for every candidate, so it cannot rank them. A multi-scale kernel statistic on compressed full datasets should stay graded. We therefore predict it will order candidates by real-domain accuracy better than the classifier, better than small-sample kernel estimates, and better than model confidence. If that holds, kernel scalability becomes useful for choosing which synthetic-trained model to deploy. We deliberately do not claim it becomes a better training loss.

Core Claim

The claim is that a near-linear-time, multi-kernel discrepancy between compressed synthetic and real feature sets works as an unsupervised model- and data-selection criterion for sim-to-real transfer. Its rank correlation with true target accuracy should exceed that of the standard label-free proxies, at a compute cost comparable to one evaluation pass over the data.

Two properties are necessary for this. First, the statistic must use the full datasets, since small subsamples are too noisy to separate candidates that differ by a few accuracy points. Second, it must combine several kernel scales without the power dilution that naive aggregation causes, because the gap spans sensor noise, texture and geometry at once.

Why this and not the obvious combination

The obvious bridge between these literatures is to train with a scaled-up alignment loss, as in DAN. DAN minimizes multi-kernel maximum mean discrepancy (MMD, the distance between the average kernel embeddings of two samples) across network layers. Gap 1 of the research file suggests running that loss on thinned coresets of the full synthetic set instead of on mini-batches. We think this is the wrong target.

The evidence from VisDA is blunt: DAN reached 51.62% on VisDA-C, while the pseudo-labeling winner reached 92.8%. Nothing in the file suggests that batch variance, rather than the alignment objective itself, caused that gap. Rethinking Joint Maximum Mean Discrepancy in fact shows that joint MMD pushes same-class points apart. It needs an independence penalty (HSIC) to stay discriminative, and it was only tested on Office-style benchmarks. ICON drops kernel distances entirely and reaches 87.4% on VisDA-2017. A faster version of a loss that loses to self-training is a weak contribution.

Measurement is different. Every one of these methods, ICON and pseudo-labeling included, still needs its hyperparameters and checkpoints chosen without target labels. Kernel two-sample statistics are designed for exactly that job: calibrated, graded comparisons of whole distributions.

Mechanism

Let $f$ be a candidate model's penultimate-layer feature map. Let $S=\{f(x^s_i)\}_{i=1}^{n}$ be the synthetic features and $T=\{f(x^t_j)\}_{j=1}^{m}$ the real features.

We borrow the compression step from Compress Then Test. That paper partitions each sample into bins, shrinks each bin with KT-COMPRESS, and computes MMD on the resulting coresets. It recovers the detection rate of the full quadratic test in near-linear time, reporting 20x to 200x speedups over approximate MMD tests without power loss. We extend it in two ways, both motivated by the structure of the sim-to-real gap.

The first extension is multi-scale aggregation. We compute $K$ statistics $\hat{\delta}_1,\dots,\hat{\delta}_K$ on the same coresets. These use Gaussian kernels at several bandwidths applied to features from two network depths, since the gap has sensor, texture and geometry layers. DUAL shows that summing or maximizing redundant kernels dilutes power. It fixes this with a covariance-aware combination, but pays for that with full quadratic U-statistics. We instead estimate the $K\times K$ covariance $\hat{\Sigma}$ from bin-level statistics that CTT's binned structure already provides, and score each candidate with

$$D = \hat{\delta}^\top \hat{\Sigma}^{-1} \hat{\delta},$$

where $\hat{\delta}$ is the vector of coreset MMD estimates. This keeps DUAL's protection against redundancy at CTT's cost. We use $D$ as a graded effect size for ranking, not as a pass/fail test.

The second extension is class-conditional comparison. Following the warning in Rethinking JMMD that marginal alignment can hide class-level mixing, we also compute $D$ separately for each class. Each comparison pairs source samples with a given label against target samples that the candidate assigns to that label, and the per-class scores are weighted by class frequency. This variant is circular in a way the marginal one is not, because it uses the candidate's own predictions. It is therefore a separately reported ablation, not the default.

A known risk constrains the design. Low-Rank Thinning proves that thinning beats uniform subsampling only when the kernel matrix's eigenvalues decay quickly. When they do not, as with near-isotropic noise, the advantage disappears. Deep features are often approximately low-rank, but this is an empirical question. We will measure the spectrum on every benchmark and report whether thinning's advantage over uniform sampling follows the decay rate, as that theory predicts.

Research Program

We will first build the compressed multi-kernel statistic as a standalone tool. We will then create candidate pools on three sim-to-real benchmarks from the file:

  • VisDA-C: about 152k CAD renderings to about 55k COCO validation images.
  • VisDA-S: GTA5 frames to Cityscapes.
  • SynLiDAR to SemanticKITTI and nuScenes.

Each pool will contain 30 to 60 trained models. They span:

  • source-only training;
  • DAN-style MMD alignment at several loss weights;
  • a pseudo-labeling/mean-teacher recipe at several confidence thresholds;
  • intermediate checkpoints of each;
  • for LiDAR, ASM-style masking models at several masking rates, following Adversarially Masking Synthetic to Mimic Real.

Target labels are used only to score candidates afterwards.

The same statistic will also be used for synthetic data selection, to test Driving in the Matrix's finding that volume and diversity drive transfer. We will pick subsets of GTA-style frames by lowest $D$ against unlabeled real frames and ask whether they beat random subsets of equal size.

Baselines

Naive reference. Select the candidate with the highest accuracy on a held-out synthetic validation split. This is the default most practitioners use. VisDA shows how badly it can mislead: AlexNet scores 99% on synthetic data and 28% on real COCO images.

Domain-classifier accuracy. Train a small classifier to separate synthetic from real features and prefer the candidate whose features it separates worst. This is the strongest simple competitor, because it is cheap, scales linearly and is widely used. Li and Kang's observation that discriminators latch onto the "clean versus noisy" shortcut is the specific reason we expect it to saturate.

Uniform-subsample quadratic MMD. Compute exact MMD on a uniform random subsample, sized so its runtime equals our method's.

Mini-batch average MMD. Average the MMD over many small batches, as DAN does, under the same runtime budget.

Confidence and entropy scores. Rank candidates by mean prediction confidence or entropy on the unlabeled target data.

Ablations. Three ablations isolate our ingredients:

  • single-bandwidth CTT versus the covariance-aware aggregate, which tests the DUAL-derived component;
  • KT-COMPRESS versus uniform coresets of equal size, which tests thinning;
  • marginal versus class-conditional scoring.

Fair comparison. All methods see the same feature extractors, the same candidate pool and the same target images. Each gets the same wall-clock budget, set to at most one forward pass over the target set. Tunable hyperparameters get an equal small grid, chosen on a separate calibration benchmark (Office-Home) and then frozen.

Metrics and the bar to beat. The main metrics are Kendall's tau between score and target accuracy, and selection regret: the target accuracy of the best candidate minus that of the chosen one. Target mean-class accuracy is used for VisDA-C and mIoU for the dense tasks. No reported result for label-free model selection on these benchmarks was found on Lacuna, so there is no published number to beat. We will instead report regret in the task's own units against the published endpoints: 51.62% (DAN) and 92.8% (winner) on VisDA-C; 21.4 and 47.5 mIoU on VisDA-S; 15.2% source-only, 22.8% ASM and 55.5% oracle mIoU on SynLiDAR to SemanticKITTI.

Beating the baselines means lower regret than the domain classifier on at least two of the three benchmarks, with non-overlapping bootstrap intervals over resampled candidate pools. For the LiDAR data we additionally require regret below the 7.6-point gap between source-only (15.2%) and ASM (22.8%), because a selector that cannot pick an adapted model over a source-only one is useless there.

Why not just…?

Why not just use a domain classifier's accuracy? This is the real threat, and the experiment is built around it. If the classifier's accuracy stays below saturation across the candidate pool, it may rank as well as our statistic at lower cost. In that case we will report that the kernel machinery is unnecessary. Our hypothesis rests on Li and Kang's finding that neural discriminators exploit the presence or absence of sensor noise rather than the semantic gap. The check is direct: plot classifier accuracy across candidates, and see whether it sits near 100% while target accuracy varies by many points.

Why not just compute exact MMD on a random subsample? Compress Then Test shows that subsampled and block estimators must grow their samples back toward quadratic cost to match full-sample power. Ranking candidates that differ by one or two accuracy points is exactly a low-signal regime. The thinning-versus-uniform ablation settles this. If the kernel spectra turn out flat, Low-Rank Thinning predicts that uniform subsampling will tie, and we will say so.

Why not just pick by target confidence or source validation accuracy? These cost nothing. However, they reward confident errors, the same confirmation bias that dooms pseudo-labeling under severe shift, where initial real-domain accuracy can be as low as 28%. We include both scores in every ranking comparison, so their failure or success is measured rather than assumed.

Why not just train with global MMD instead of using it for selection? DAN's 51.62% against the winner's 92.8% on VisDA-C, and the opposite-signs result in Rethinking JMMD, give no reason to expect that a better-estimated alignment loss closes that gap. We add one cheap arm that trains DAN with the compressed statistic as its loss. If it moves VisDA-C accuracy substantially, that would redirect the program.

Decisive Checks

The central experiment is the rank-correlation and regret comparison on the VisDA-C candidate pool. Within that pool, the most informative single outcome is whether the covariance-aware aggregate beats single-bandwidth CTT. That comparison tests whether the gap's multi-scale structure matters for measuring it.

A second check is whether thinning's advantage over uniform coresets tracks the measured eigenvalue decay of the feature kernel across the three modalities, as Low-Rank Thinning predicts.

A third is the data-selection test. Frames chosen for the lowest $D$ should beat equally sized random subsets on real vehicle detection or segmentation. If they instead lose, that would support Driving in the Matrix's view that diversity matters more than closeness to the target.

Negative results on any of the three are publishable, because each rules out a specific use of scalable kernel tests in sim-to-real practice.