A Learned LiDAR Imperfection Field for Synthetic-to-Real Segmentation

Simulated LiDAR scans are too clean. A simulator returns a point for every beam that hits a surface. A real sensor loses points on glass, wet asphalt and dark paint. It thins out with distance, has dead or weak beams, and has sectors blocked by the vehicle itself. A 3D segmentation network trained on clean synthetic scans learns to expect this completeness and fails on real scans.

This proposal learns the missing imperfections directly from unlabeled real scans. It does so with a compact field that maps sensor-frame coordinates and surface properties to two outputs: the probability that a point is lost, and how much its measured range is perturbed. The field is stored as a multi-resolution hash encoding, meaning a stack of coarse-to-fine grids whose corner features live in small learnable lookup tables and are decoded by a tiny network. This structure can represent both slow trends, such as distance falloff, and very sharp localized structure, such as a single dead beam or a reflective window, at low memory cost. The degraded synthetic scans then train an ordinary segmentation network.

Core Claim

The claim is that much of the remaining synthetic-to-real gap in LiDAR segmentation comes from sensor degradation with a specific shape. It is multi-scale. It is sharp along the beam (elevation) axis. It depends on what the surface is made of and the angle at which the beam hits it.

An explicit, continuous imperfection field can capture this better than the binary masking used today. That field lives in a space of roughly five to six dimensions: beam elevation, azimuth, range, incidence angle, and the synthetic semantic class as a material proxy.

The closest prior work is Adversarially Masking Synthetic to Mimic Real (ASM). ASM learns Gumbel-Softmax binary drop masks inside the shallow layers of the segmentation backbone. It reaches 22.8% mean intersection-over-union (mIoU) on SynLiDAR to SemKITTI, against 15.2% for source-only training and 55.5% for a real-data oracle.

Our proposal differs from ASM in three ways:

  1. The degradation is a standalone sensor model in input space, not a masking layer tied to one backbone. It can therefore be inspected, reused across backbones, and re-targeted to a different sensor.
  2. It models continuous range perturbation as well as dropout.
  3. It is explicitly conditioned on surface material and incidence angle, which are the physical causes of dropout described in the ASM paper itself.

The multi-resolution hash encoding is chosen deliberately, and the experiments are set up to falsify that choice. Kim et al. showed that at matched parameter budgets, plain interpolated grids beat hash encodings on dense signals in two and three dimensions. Hash encodings win only when the signal sits on a sparse lower-dimensional structure.

Our input space is five- to six-dimensional, where dense grids become unaffordable. The degradation signal is also concentrated: it is nearly uniform almost everywhere, with sharp exceptions on particular beams and materials. That is the regime where bounded hash tables put their capacity on the informative regions, as described by Zha et al.. Audia et al. give a second reason. Their analysis shows that grid encodings raise the smallest learnable-frequency eigenvalue by about eight orders of magnitude over plain multilayer perceptrons. A plain network should therefore blur exactly the sharp beam-level structure we care about. If a dense grid over the three sensor-frame dimensions, or a plain network over all inputs, matches the hash field, we will report that and drop the hash-encoding claim.

Research Program

Each synthetic point has a beam elevation $e$, an azimuth $a$, a range $r$, an incidence angle $\theta$ (computed from the synthetic surface normal) and a semantic label $c$. The imperfection field $F$ maps $(e, a, r, \theta)$ through a multi-resolution hash encoding. It concatenates a learned embedding of $c$ and decodes with a two-layer network into a keep probability $p$ and a range offset distribution with mean $\mu$ and scale $\sigma$.

Degraded scans are sampled with a straight-through Gumbel relaxation for keep/drop and reparameterized noise for range. They are then re-projected into a range image.

A small discriminator compares range-image patches from degraded synthetic scans against patches from unlabeled real scans of the target sensor. The field is trained to fool it. A penalty on the total drop rate prevents the trivial solution of deleting whole objects.

The segmentation network is trained on degraded synthetic scans with their labels. Optionally, ASM's feature-level masking is kept on top, to test whether the two are complementary.

Two lessons from hash-grid optimization shape the training schedule:

  • Smoothness. Wang et al. found that analytical gradients through hash grids update isolated table entries and produce noisy fields. A degradation field with salt-and-pepper texture would itself be a new simulator artifact. We therefore add a finite-difference smoothness penalty along azimuth and range, but not along elevation, where sharp structure is expected.
  • Coarse-to-fine schedule. We use the coarse-to-fine level schedule of Heo et al., so that broad trends such as range falloff settle before fine levels can encode beam-specific detail.

Heo et al.'s "jiggling" gradient problem concerns derivatives with respect to input coordinates. Our inputs are fixed by the synthetic geometry and only the tables are optimized, so that failure mode does not apply here.

For re-targeting to a different sensor, such as the 32-beam nuScenes LiDAR, synthetic scans are first resampled to the target beam layout. A field is then fit on unlabeled target scans. This tests whether the learned structure reflects the physics of each sensor or merely memorizes one dataset.

Baselines

Naive reference. The naive reference is source-only training on clean SynLiDAR. ASM reports 15.2% mIoU on SemKITTI and 11.5% on nuScenes for this setting.

Hand-tuned randomization. We add a randomization baseline that applies uniform random dropout and Gaussian range jitter. Its rates are tuned on the same small labeled validation split that every method may use. This represents what a practitioner would do without learning anything.

Strongest existing method. The strongest existing method is ASM, at 22.8% mIoU on SemKITTI and 17.0% on nuScenes. We run ASM with the same segmentation backbone, the same unlabeled real scans, the same training schedule and an equal hyperparameter search budget, rather than relying only on quoted numbers. CoSMix, reported at 20.0% on SemKITTI in the same paper, is included as a mixing-based adaptation reference.

What counts as success. Beating ASM means exceeding its re-run mIoU on both target datasets by more than the spread across three random seeds. Matching ASM while adding transfer across backbones and sensors would be a weaker but still useful result. Falling short of ASM on both datasets would count as a failure.

Ablations. Four ablations isolate the key ingredients:

  1. The same objective with a plain multilayer perceptron in place of the hash encoding.
  2. A dense trilinear grid over $(e, a, r)$ at matched parameter count, with total-variation regularization, which is the strong competitor identified by Kim et al.
  3. The hash field without the material and incidence inputs.
  4. The hash field restricted to binary dropout without range perturbation.

Why not just…?

Why not just keep ASM's in-network masking? ASM is the real threat, and it already learns dropout adaptively. Its masks, however, live inside one backbone's shallow features. They are binary, and they have no explicit access to material or incidence angle. If our field only matches ASM, the remaining argument is portability, and we test it directly: a field trained once is reused, unchanged, to train a different segmentation backbone. ASM must be retrained for each backbone. We also run ASM combined with our field. If combining them adds nothing over ASM alone, the field is redundant and we will say so.

Why not just fit a parametric physical dropout model? A hand-designed model would capture the probability of a return as a function of range and incidence angle, with per-class reflectance constants. It is cheaper and more interpretable than our field. It cannot represent dead beams, self-occlusion sectors or other quirks of a particular sensor unit unless someone writes them in by hand. We include it as a baseline fit by the same adversarial objective. Where it falls short, the per-beam return-rate residual maps should show beam-specific structure that it misses and the hash field captures.

Why not just a dense grid or a small plain network? Kim et al. make this a serious question, because dense grids win at matched budgets in low dimensions. Ablations 1 and 2 answer it directly. Our prediction is specific: the dense grid will match the hash field when material and incidence inputs are dropped, but cannot afford them. The plain network will under-fit beam-level structure, which will be visible as a larger per-beam return-rate error.

Why not just randomize dropout broadly? Neural-Sim found, for image rendering, that optimizing simulation parameters against real data beat untargeted generation by 20% to 60% in detection accuracy. That is suggestive but comes from a different modality. Our randomization baseline tests the claim directly for LiDAR.

Proposed Experiments

The main benchmarks are SynLiDAR to SemKITTI and SynLiDAR to nuScenes, scored by per-class and mean IoU under the ASM protocol. Sensor fidelity is measured independently of the downstream task. We compare the return rate of degraded synthetic scans and real scans for each (beam, range bin, class), using Earth mover's distance between the two return-rate distributions.

Per-class analysis tests whether the material conditioning matters where physics says it should. Gains should concentrate on classes with glass, dark surfaces or road, rather than spreading uniformly.

There are four decisive checks:

  1. SemKITTI result. The full field must beat re-run ASM on SemKITTI.
  2. Input-space advantage. The field must beat the dense-grid and plain-network ablations on both mIoU and return-rate distance. Without this, the hash encoding is not justified.
  3. Backbone reuse. A field trained with one backbone must improve a second backbone without retraining.
  4. Sensor specificity. A SemKITTI-fit field applied to nuScenes must underperform a nuScenes-fit field. This shows that it learned sensor-specific structure rather than generic noise.

There is also an honest kill criterion. The field may close the return-rate statistics almost completely while mIoU gains stay small compared with the 22.8% to 55.5% oracle gap. That outcome would show that the remaining gap is mainly scene content and geometry rather than sensor imperfection. Such a negative result is worth reporting, because it would redirect effort away from noise modeling altogether.