Training Geometry-Only Networks to Ignore Sampling Density Before They See Real Sensors

Models that see only 3D shape, with no color or texture, often do better than models that also see RGB images. A detector that reads only point coordinates beat RGB-D detectors on indoor benchmarks. A robot hand policy that saw only point clouds grasped novel real bottles more often than it did in simulation. This suggests geometry should cross the gap between simulated and real data easily, since shape does not depend on lighting. Yet it does not. A LiDAR segmentation network trained on synthetic scans and tested on real SemanticKITTI scans reaches 15.2% mean intersection-over-union (mIoU). The best learned adaptation reaches 22.8%. A model trained on real labels reaches 55.5% (Adversarially Masking Synthetic to Mimic Real).

We argue that much of this gap comes from where the sensor happens to place its points, not from the shape itself. How densely points land, how density falls off with distance, and which laser rings drop out are properties of the sensor. A geometry network is free to learn them as a shortcut. We propose to measure that shortcut directly. We then propose to remove it during synthetic training by making the network's features agree across many physically parameterized resamplings of the same synthetic scene. The method needs no real target scans at any point. If it works, a geometry-only model trained purely in simulation should close a substantial part of the oracle gap without ever looking at the target sensor.

Core Claim

Current point-cloud sim-to-real methods try to imitate the target sensor. Adversarial Spatial Masking learns to drop synthetic points until a discriminator cannot tell them from real SemanticKITTI scans (Adversarially Masking Synthetic to Mimic Real). SynLiDAR's Point Cloud Translator learns the real beam-dropout pattern through a range-image translation network (SynLiDAR). SIRA-PCR uses a generative adversarial network (GAN) to resample synthetic points toward a real sensor profile, with each moved point kept inside the convex hull of its neighbors (SIRA-PCR). All three need unlabeled target data, and all three produce a model tuned to one sensor.

Our claim is the opposite design choice. For geometry-only networks, invariance to sampling, taught on synthetic data alone, transfers at least as well as imitation of the target sensor. It should also transfer better to sensors never seen in training.

The method reuses two ideas that were developed for other problems and have never been used in sim-to-real training:

  • Group-wise contrastive learning comes from Density-invariant Features for Distant Point Cloud Registration. That work showed that pairwise training objectives fall into "density shortcuts" when matching real LiDAR scans taken at different distances. It fixed this with a group-wise contrastive objective across several views, improving KITTI registration recall in the 40–50 m range by 40.9% over Predator. It did not study synthetic-to-real transfer.
  • Neighborhood normalization comes from Neighborhood Normalization for Robust Geometric Feature Learning. That work showed that standard normalization layers break under resolution mismatch and that local neighborhood normalization restores matching. It likewise did not study synthetic-to-real transfer.

Applying these invariance tools to synthetic training, rather than to real-to-real matching, is what is new.

Research Program

The program has three parts: a diagnosis, a method, and a set of tests.

Diagnosis

Before building anything, we test whether density is actually the shortcut. Doing this first lets the method stand or fall on evidence.

We train a source-only segmentation network on SynLiDAR using coordinates only, with no intensity channel. We then run three measurements:

  • Density probe. We fit a linear classifier on the network's frozen per-point features to predict the local point density and the laser ring index. High probe accuracy means the network encodes sampling rather than shape alone.
  • Density harmonization at test time. We resample both synthetic and real test scans onto a common, distance-independent voxel density. We then measure how much of the source-only mIoU drop on SemanticKITTI disappears. The file does not constrain the size of this effect; the experiment decides it.
  • Reverse check. We take the real-data oracle and test it on real scans that have been resampled with synthetic-style density. This tells us whether real-trained models are equally fragile to sampling changes.

If harmonization recovers a large share of the gap, the density-shortcut hypothesis is supported. If it recovers little, the remaining deficit lies elsewhere. The two leading candidates are shape differences between synthetic and real objects, which is what SynLiDAR's appearance module targets, and missing physical properties such as incidence angle. In that case we narrow the proposal to a diagnostic paper rather than force the method.

Method

For each synthetic scan, we render $K$ resampled views from a family of physically parameterized sampling processes, with no target data. The family covers:

  • quadratic density decay with distance, the mechanism documented for real LiDAR in Density-invariant Features;
  • beam counts spanning 32 to 128 rings;
  • random ring dropout;
  • range-dependent angular jitter.

The backbone uses neighborhood normalization in place of batch normalization. We train with the supervised segmentation loss on every view plus a group-wise contrastive term over the $K$ views. The contrastive term pulls together the features of points that share the same underlying surface location across views and pushes apart features from different locations. The objective is

$$ \mathcal{L} = \frac{1}{K}\sum_{k=1}^{K} \mathcal{L}_{\text{seg}}(x_k, y_k) + \lambda\, \mathcal{L}_{\text{group}}(f(x_1),\dots,f(x_K)), $$

where $x_k$ is the $k$-th resampled view with labels $y_k$, $f$ is the per-point feature extractor, and $\lambda$ weights the invariance term.

The sampling family is deliberately broad rather than fitted to any real sensor. This breadth is what separates the method from target-imitation approaches.

Tests

We evaluate on SynLiDAR→SemanticKITTI and SynLiDAR→nuScenes segmentation. We also run a second geometry task to check that the effect is not specific to segmentation: FlyingShapes→3DMatch, 3DLoMatch, and ETH registration, using the SIRA-PCR protocol.

Baselines

Source-only. The naive reference point trains the same backbone on clean SynLiDAR and tests on real scans. Adversarially Masking Synthetic to Mimic Real reports 15.2% mIoU on SemanticKITTI and 11.5% on nuScenes for this setting. We rerun it with our backbone so every comparison shares one architecture.

Adversarial Spatial Masking (ASM). This is the strongest existing competitor on exactly this question. It learns, adversarially against unlabeled real scans, which synthetic points to drop. It reports 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes. The same paper reports 20.0% for both CoSMix and SqueezeSegV2 on SemanticKITTI.

For a fair comparison, we run ASM with our backbone, the same SynLiDAR training split, the same number of training steps, and the same hyperparameter search budget. We report both our rerun and the published figure. ASM sees unlabeled target scans and our method does not, so matching ASM already counts as a result. Beating it on SemanticKITTI by more than seed variance, while also beating it on nuScenes, counts as a clear win. The nuScenes test is especially informative for a model never tuned to the nuScenes sensor. A further check trains ASM against SemanticKITTI and tests it on nuScenes. The claim predicts that imitation of one sensor transfers poorly to another, while invariance does not.

SynLiDAR's Point Cloud Translator. SynLiDAR reports improving SemanticKITTI mIoU from 60.3% to 64.7%. The file does not state that protocol clearly enough to compare directly; it appears to involve real labels. We therefore treat PCT's sparsity module as a baseline we rerun in the unsupervised setting rather than cite a number for it.

Registration baseline. On registration, the strongest baseline is SIRA-PCR with a GeoTransformer backbone. It reports 94.1% registration recall on 3DMatch and 76.6% on 3DLoMatch, and a 1.9-point zero-shot gain on ETH. The file gives no absolute ETH figure for SIRA-PCR.

Oracle. The real-data oracle, 55.5% mIoU on SemanticKITTI, is the ceiling. We report results as the fraction of the gap between source-only and oracle that each method closes.

Ablations. The ablations isolate the key ingredient in three steps:

  • the same multi-view resampling used as plain augmentation, without the contrastive term;
  • the contrastive term with batch normalization instead of neighborhood normalization;
  • a sampling family fitted to SemanticKITTI statistics instead of kept broad.

Why not just…?

Why not just augment synthetic scans with random point dropout or distance-based downsampling? This is the real threat. The "augmentation only" ablation above is exactly this alternative, run under the same resampling family and the same compute. Pure Gaussian jitter is known to fail because real sensor noise is structured and depends on the signal (Quasi-Balanced Self-Training on Noise-Aware Synthesis). Physically parameterized downsampling, however, is not ruled out by anything in the file. The contrastive term earns its place only if it beats this augmentation on the density probe and on target mIoU. If plain augmentation matches it, we will report the simpler recipe as the finding. The paper then becomes the density diagnosis plus "use broad physical resampling," which is still useful.

Why not just use ASM plus self-training? ASM's authors note they never combined it with self-training, so that combination may well beat us on SemanticKITTI. However, it still requires unlabeled target scans and fits itself to one sensor. We will test our method as a starting point for ASM and for self-training. If invariance is complementary to imitation, the combination should exceed either alone. The cross-sensor test separates the two approaches regardless of that outcome.

Why not just add RGB or intensity to fill in what geometry misses? Manipulation as in Simulation shows that RGB-guided depth cleaning can rescue sim-trained policies. Real success went from 0/30 to 26/30. The cost is reintroducing a dependence on lighting and texture. SynLiDAR does not simulate intensity, so intensity cannot be trained in this setting at all. Evidence that geometry alone can outperform RGB-D fusion comes from VoteNet, with 58.6 mAP on ScanNetV2 and 57.7 on SUN RGB-D, and from TransPose, with 99.4% ADD-S on LineMod. Both results are within-domain, so they motivate rather than settle our question. We keep the modality fixed to coordinates so that any improvement is attributable to sampling invariance.

Proposed Experiments and Decisive Checks

The first decisive check is the diagnosis. Two results support the hypothesis: a high density-probe accuracy for the source-only model, and a meaningful mIoU recovery under test-time density harmonization. If neither holds, the method is not pursued.

The second check is whether our target-free model reaches ASM's 22.8% on SemanticKITTI and 17.0% on nuScenes, and exceeds them when rerun under a matched budget.

The third check is the cross-sensor test. Imitation-based methods trained against one real sensor should lose ground on the other sensor. Our model should not. The density probe should show that our features carry little sampling information.

The fourth check is whether the gains carry over to registration. We compare against SIRA-PCR on 3DMatch, 3DLoMatch, and ETH, replacing its target-profiled GAN resampling with our invariance training.

All results are reported over three seeds, as the fraction of the source-to-oracle gap closed. Per-class results are split by distance bin, because the hypothesis predicts the largest gains on distant, sparsely sampled objects.