Do Self-Supervised Sim-to-Real Gains Survive a Placebo Objective?

Perception models trained in simulation often fail on real sensors. The standard remedy is self-supervised adaptation: fine-tune on unlabeled real scans using a loss built from a physical rule, such as "objects move as rigid pieces" or "a completed shape should look the same from every viewpoint." Papers report that these physical losses close part of the simulation-to-reality gap on fixed real benchmarks like SemanticKITTI. Increasingly, though, the adapted network starts from a backbone pretrained on large unlabeled sensor or web collections, and those collections may already contain the benchmark's own scans. In language models, Reasoning or Memorization? showed what this does: reinforcement learning with random or even inverted rewards still improved Qwen2.5 on the leaked MATH-500 benchmark, because any training signal helped the model recall text it had already seen. On a freshly generated arithmetic set the same spurious rewards failed.

We propose the perception version of that test. Alongside every self-supervised adaptation run, we run a placebo: the same pipeline, compute, and data, but with a physically meaningless version of the loss. If the placebo recovers most of the benchmark gain, the physical prior is not what is closing the gap. We then use controlled pretraining, with known amounts of benchmark leakage deliberately injected, to measure how much leakage inflates reported sim-to-real gains and whether the placebo gap can detect it. The deliverable is an evaluation protocol that can be applied to published pipelines and that separates "the physical rule adapted the model" from "adaptation reactivated what the backbone already knew about the test scenes."

Core Claim

Our claim is that the gain from a self-supervised physical loss is only interpretable relative to a placebo loss run under identical conditions. We further claim that, when a backbone has seen unlabeled scans from the target benchmark, the gap between the physical loss and the placebo will shrink on that benchmark but not on an unexposed real target. This yields a diagnostic that needs no access to the pretraining corpus, and a corrected gain (physical minus placebo) that should rank adaptation methods the same way whether or not the benchmark leaked.

This is not the RLVR paper transplanted wholesale. That work studies reward signals for text generation, and it detects leakage with prefix-completion probes on token sequences. Here the spurious signal is a corrupted geometric loss, the leakage is unlabeled sensor data rather than question–answer text, and the question is not "is the model contaminated?" but "which part of a reported adaptation gain is due to the method?" The research file identifies no paper that combines self-supervised sim-to-real adaptation with contamination controls. Its Gap 5 asks exactly this question, whether self-bootstrapping can trigger retrieval the way spurious rewards do, and leaves it open.

Why this matters now

Three findings make the question pressing. First, the sim-to-real adaptation literature treats its real targets as clean, fixed ground truth. ASM learns where to drop points in synthetic LiDAR so that it resembles real LiDAR, and it improves SynLiDAR→SemanticKITTI segmentation from 15.2% to 22.8% mean intersection-over-union (mIoU, the average overlap between predicted and true class regions), against an oracle of 55.5% trained on real labels. RigidFlow generates motion pseudo-labels by fitting rigid transforms to small point clusters, and ACL-SPC completes shapes by requiring the same output from re-projected partial views. None of these works asks what the starting weights already encode about the target scenes. Second, the newest pipelines explicitly rest on web-scale backbones: Driving with DINO converts simulator video to realistic nuPlan-style video using DINOv3 features, and it does not audit whether nuPlan-like footage appeared in DINOv3's pretraining. Third, S2R-Bench shows that our picture of real degradation is unreliable in the other direction too: real high-glare LiDAR performance was 85% lower than simulated corruptions predicted. Benchmark numbers in this area are therefore interpreted with little independent calibration.

The contamination literature supplies the method. Dynamic generators such as DyVal and RandomCalculation remove leakage by generating fresh test items, but the file notes that this has not been extended to perception. We do not try to procedurally generate real sensor data, which would reintroduce the sim-to-real gap. Instead we borrow the other half of the RLVR paper's design: a spurious-signal control, compared across a leaked and an unleaked target.

Research Program

The first component defines placebo losses for each adaptation method. A placebo must be as similar as possible to the real loss in gradient magnitude, number of update steps, and which parameters it touches, while removing the physical content. For RigidFlow, the placebo fits rigid transforms to randomly permuted cluster assignments, so each "rigid piece" is a scatter of unrelated points; the pseudo-labels have realistic scale but no physical meaning. For ASM, the placebo replaces the learned drop mask with a random mask matched to the learned mask's drop rate, which separates "learned realistic dropout" from "any dropout." For an ACL-SPC-style viewpoint consistency loss, the placebo pairs each prediction with a view re-projected from a different object. We also include the analogue of the RLVR paper's inverted reward: a loss that rewards violating the prior, such as maximizing rigidity residuals. We rescale each placebo to match the real loss's gradient norm at initialization, so that effects of optimization strength alone cancel.

The second component is controlled leakage, which gives us ground truth that published backbones cannot. We pretrain a LiDAR segmentation backbone with masked-point reconstruction on a pooled corpus of unlabeled driving scans, and we build three versions of the corpus. The clean version excludes every KITTI-derived scan. The adjacent version includes unlabeled KITTI raw scans from the SemanticKITTI training sequences, the same city and sensor, but excludes validation sequence 08. The direct version additionally includes the unlabeled scans of sequence 08. The direct version is realistic, because unlabeled raw scans are public and easy to crawl. nuScenes is excluded from all three corpora, which makes SynLiDAR→nuScenes an unexposed real target with a different sensor. Starting from each backbone, we run source training on SynLiDAR, then either the physical adaptation loss or its placebo. We evaluate on sequence 08 and on nuScenes. We repeat the design for scene flow, using FlyingThings3D as the source, KITTI as the leaked target, and an excluded real sequence set as the unexposed target.

The third component applies the protocol to published pipelines whose backbones' exposure is unknown. We run placebo versions of the adaptation stages built on public pretrained backbones and report the physical-minus-placebo gap on each benchmark. We do not claim contamination from this alone. We claim it only when the pattern matches the one calibrated in the controlled experiment: a small gap on the benchmark and a large gap on the unexposed target.

The main quantity is the corrected gain for method $m$ on target $t$:

$$ G_{\text{corr}}(m,t) = A_{\text{phys}}(m,t) - A_{\text{placebo}}(m,t), $$

where $A_{\text{phys}}$ is the accuracy after adaptation with the physical loss and $A_{\text{placebo}}$ is the accuracy after the matched placebo. Both use the same backbone, data, and step budget, with mIoU for segmentation or endpoint error for flow. Our leakage signature is the ratio of $G_{\text{corr}}$ on the leaked target to $G_{\text{corr}}$ on the unexposed target. For each target, we first normalize the gain by that target's source-only-to-oracle range, so that targets of different difficulty are comparable.

Baselines

The naive reference point is source-only training: the SynLiDAR-trained model applied directly to real scans, with no adaptation. ASM reports 15.2% mIoU for this on SemanticKITTI and 11.5% on nuScenes. We rerun source-only from each of our three backbones, because leakage may already raise the source-only number before any adaptation happens. That effect is itself a finding.

The strongest existing method in this setting is ASM, which reports 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes. It beats CoSMix and SqueezeSegV2, both at 20.0%, and the oracle is 55.5%. We rerun ASM with its published training schedule from each of our backbones, with the same data, the same number of GPU-hours, and an identical hyperparameter search budget. The placebo gets exactly the same budget. Beating ASM is not the goal of this project. The relevant test is whether ASM's gain over the placebo holds up. The protocol would be vindicated if ASM's corrected gain on sequence 08 drops by at least half from the clean backbone to the direct backbone, while its corrected gain on nuScenes changes by less than the seed-to-seed standard deviation. For scene flow, the reference is RigidFlow, which reports 0.069 m endpoint error on FlyingThings3D. For its KITTI generalization, no reported result was found on Lacuna, and the file only states that it outperformed synthetic-trained supervised models. We will therefore establish our own KITTI numbers for RigidFlow and its placebo.

The key ablation isolates the placebo's content from the mere act of updating weights on target data. We run adaptation that updates only normalization statistics on target scans with no loss at all, which is the cheapest form of test-time adaptation discussed at venues like the Test-Time Updates workshop. No reported result for this baseline on SynLiDAR→SemanticKITTI was found on Lacuna. If normalization-only adaptation matches the placebo, the placebo is measuring statistic re-estimation and nothing more. A second ablation varies the placebo's gradient-norm matching to show that the conclusions do not depend on how strongly the placebo pushes the weights.

Why not just…?

Why not just evaluate on a newer real dataset that the backbone cannot have seen? This is the most serious alternative, and in part we adopt it: nuScenes plays that role in the controlled study, and S2R-Bench is a natural fresh target for detection-style checks. The limitation is that a new dataset changes the city, sensor, and weather at the same time as removing leakage, and S2R-Bench shows that real shifts can be far larger than expected. A drop on fresh data therefore cannot tell leakage apart from ordinary domain shift. The placebo comparison is made within a single benchmark, so the domain is held fixed. The controlled experiment decides the matter: if the leaked-versus-unexposed performance gap alone already recovers the injected leakage level as well as the placebo ratio does, the placebo adds nothing and we will report that.

Why not just search the pretraining corpus for benchmark scans? When the corpus is available, this should be done, and we do it for our own corpora. However, the backbones in current pipelines, such as DINOv3 in Driving with DINO, do not come with auditable corpora, and the audit tools in the file, such as the Data Contamination Quiz, are built for text. Retrieval also misses the adjacent-exposure case, where the backbone saw neighboring frames from the same streets but no exact test frame. Our adjacent corpus measures how much that case inflates gains when no duplicate exists to find.

Why not just probe memorization directly, for example by masking part of a test scan and checking whether the backbone fills it in unusually well? This is the visual counterpart of partial-prompt completion, and it will probably detect exposure. We will run it on all three backbones and expect it to rank them correctly. It answers a different question, though. It tells us the backbone saw the scans, but not how much of a particular adaptation method's reported gain depends on that exposure. A backbone could be exposed while a method's corrected gain is unaffected, or the reverse. We report both signals and their correlation. If masked reconstruction predicts the size of the gain inflation as well as the placebo ratio does, the cheaper probe should become the recommended tool.

Why not just treat normalization re-estimation as the whole story? It may explain the placebo gains on unexposed targets. The normalization-only ablation above tests this directly, and the leakage question stands only if the placebo's advantage over normalization-only grows as leakage increases.

Proposed Experiments

The decisive experiment is the grid of three backbones (clean, adjacent, direct), two losses (physical and placebo), and two targets (SemanticKITTI sequence 08 and nuScenes), run for ASM and for a RigidFlow-style scene-flow pipeline with five seeds each. The prediction that would support the proposal has three parts. First, placebo gains on sequence 08 rise with leakage level. Second, corrected gains on sequence 08 fall with leakage level. Third, corrected gains on nuScenes stay flat. If placebo gains are flat across leakage levels, leakage does not drive self-supervised adaptation gains in this regime. That would be a useful negative result, and it would clear the existing benchmarks of this concern.

The second check is ranking stability. We compare source-only, normalization-only, CoSMix-style mixing, ASM, and a RigidFlow-style prior, ranking them by raw gain and by corrected gain under each backbone. We then compute rank correlation between the clean-backbone and direct-backbone rankings. The protocol is useful if corrected gains preserve the ranking while raw gains reorder it.

The third check is transfer of the diagnostic to public backbones. For pipelines built on public pretrained features, we report the leakage signature and test whether it falls in the range calibrated by the controlled study. We report these results as calibrated evidence, not proof. We also release the placebo loss implementations so that future sim-to-real papers can report corrected gains alongside raw ones.