Where to Drop Synthetic Points: Adaptive Partitions for LiDAR Sim-to-Real Transfer

Laser scanners (LiDAR) on real cars return far fewer points than their simulated counterparts, and they lose points in a structured way. Returns thin out with distance. They vanish on glass and dark paint, and they weaken on surfaces hit at grazing angles. A segmentation network trained on clean synthetic scans learns to rely on point density that never appears on the road. The most effective fix so far does something very plain: it throws synthetic points away until their density matches the real sensor's. Density-guided Translator Boosts Synthetic-to-Real Unsupervised Domain Adaptive Segmentation of 3D Point Clouds (DGT-ST) does this. It slices space into fixed concentric rings around the sensor and drops points in each ring according to the synthetic-to-real density ratio. With self-training on top, this lifts SynLiDAR→SemanticKITTI to 43.1% mean intersection-over-union (mIoU, the per-class overlap score averaged over classes), well above earlier methods. The rings assume distance is the only thing that controls how many points survive, and real sensors violate that assumption.

We propose to keep DGT's "discard, don't generate" principle and change where the discard rate is estimated. The keep probability would be learned over a small space of sampling conditions: distance, beam elevation, the angle at which the beam strikes the surface, and the surface's semantic class. That space would be partitioned adaptively, the way density-aware trees partition skewed geographic data. A cell splits only where the real data contain enough points to estimate the rate reliably and where the rate actually changes. The result is a non-learned, stable translator that is fine where reality is complicated and coarse where data are scarce. It can be dropped into existing adaptation pipelines.

Core Claim

The claim has two parts, and the experiments separate them. The first part is about what to condition on. The real-to-synthetic survival rate of a point depends strongly on incidence angle and surface class as well as range. Matching density along these extra axes should help the classes DGT-style methods still miss, such as thin, distant, and reflective objects like poles, fences, people, and car windows. The second part is about how to estimate it. Once the conditioning space has four or more axes, a fixed grid runs into the resolution dilemma described in the adaptive partitioning literature. Fine cells are starved of real points and give noisy rates. Coarse cells average away the structure. A partition that trades estimation variance against within-cell non-uniformity should beat any fixed grid at equal tuning budget.

This tradeoff has a direct precedent. Differentially Private Grids for Geospatial Data showed that a two-level adaptive grid wins by balancing noise error against the error of treating a cell as uniform. In our setting the noise comes from finite target samples rather than injected privacy noise, but the structure of the tradeoff is the same.

What Each Prior Line Contributes

Three LiDAR adaptation methods motivate the problem, and each leaves a specific opening. DGT-ST establishes that input-level density matching is the strongest single lever, but its partition is one-dimensional and fixed. Its official code (yuan-zm/DGT-ST) also reports that the adversarial warm-up stage (PCAN) cannot reproduce the published numbers because GAN training is unstable. We therefore avoid adversarial components entirely. Adversarially Masking Synthetic to Mimic Real (ASM) learns point-wise drop decisions against a domain discriminator. This shows that structured dropout matters, but it reaches only 22.8% mIoU on SynLiDAR→SemanticKITTI, and its drops are not tied to any interpretable physical condition. SynLiDAR supplies the source data and frames the gap as appearance plus sparsity. Its sparsity translator works through 2D range images with a GAN, which brings projection artifacts and instability that our 3D statistical estimator avoids.

The method comes from the partitioning literature. Spatial properties of Bayesian unsupervised trees shows that recursive trees adapt to spatially heterogeneous densities at near-minimax rates and ignore irrelevant dimensions automatically. That second property matters here: if incidence angle turns out not to matter for some class, the tree should not split on it. Their result that size-penalized priors handle spiky local variation better than depth-penalized ones informs our choice of stopping rule. PQMass uses density-adaptive Voronoi cells and a chi-squared test to compare two sample sets without a learned feature extractor. We borrow that test as a split criterion and as an evaluation diagnostic. Its warning that per-cell tests become unreliable with few samples is exactly the constraint our minimum-count rule enforces.

Research Program

The translator has three stages.

The first stage computes, for every point in both domains, a sampling-condition vector. Its components are horizontal range, beam elevation angle, incidence angle (from a local surface normal fitted to neighbors), height above an estimated ground plane, and semantic class. Class comes from ground truth for synthetic points and from teacher pseudo-labels for real points. For each point we also estimate local surface density, meaning points per unit surface area, from the radius containing its $k$ nearest neighbors. This normalizes away the fact that the two domains contain different amounts of each kind of surface, a confound that raw per-ring point counts in DGT do not remove.

The second stage grows a tree over the condition space. A node holds the median surface density of real points $\rho_T$ and of synthetic points $\rho_S$ that fall inside it. Its keep probability is $p = \min(1, \rho_T/\rho_S)$. A node splits along a candidate axis only under two conditions. Both children must contain at least $m$ real points. The children's density distributions must also differ significantly, judged by a PQMass-style chi-squared test on binned density values corrected for the number of candidate splits. The minimum-count rule sets the variance floor, the test controls non-uniformity, and $m$ and the significance level are the only knobs.

The third stage applies the translator. Each synthetic point is kept with its leaf's probability. Additionally, for leaves where real data show near-total loss (glass, very dark surfaces at long range), whole surface patches are dropped together rather than as isolated points, which mimics how real dropout looks. The tree is refit when the teacher's pseudo-labels are refreshed. We also build a class-agnostic variant that omits the class axis, so that the method's dependence on pseudo-labels can be measured rather than assumed.

The translator replaces DGT inside an otherwise unchanged self-training loop: a teacher-student pipeline with LaserMix-style consistency, as in DGT-ST's SAC-LM stage. The same leaf probabilities can also resample synthetic objects before they are pasted into real scenes in CoSMix-style mixing. CoSMix currently pastes objects at synthetic density, which creates exactly the kind of density seam a network can use as a shortcut. We treat this as a secondary experiment.

Baselines

The naive reference point is source-only training on SynLiDAR, which ASM reports at 15.2% mIoU on SemanticKITTI and 11.5% on nuScenes. We will rerun it in our backbone, the MinkowskiEngine sparse network used in the DGT-ST release. Next is global random downsampling to the real domain's overall point count. It controls for the possibility that any thinning helps.

The strongest existing method is DGT-ST, which reports 43.1% mIoU on SynLiDAR→SemanticKITTI and 50.8% on SynLiDAR→SemanticPOSS. For nuScenes we found no reported DGT-ST result on Lacuna; the best reported number there is ASM's 17.0%. Because PCAN does not reproduce, we will report the published DGT-ST numbers alongside our own rerun from the official code. Our main comparison then swaps only the translator: DGT rings versus our tree, with the same warm-up (the code's non-adversarial warm-up), same SAC-LM self-training, same schedule, and same number of GPU hours. Hyperparameters get the same budget on both sides: twenty configurations each, chosen on a held-out validation sequence. For DGT that covers ring count and jitter; for us it covers $m$ and the significance level. Beating DGT means a gain in mean mIoU over three seeds that exceeds the seed spread on at least two of the three targets, with gains concentrated in the classes and ranges the hypothesis predicts.

Two ablations isolate the key ingredient. The first is a fixed grid over the same condition space, with resolution tuned under the same twenty-configuration budget. It separates "conditioning on more variables" from "adaptive estimation." The second is our tree restricted to the range axis alone, which tests whether adaptivity helps even when the conditioning is the same as DGT's. We will also run CoSMix (reported 32.2 on SemanticKITTI, 40.4 on SemanticPOSS) with and without density-matched pasting.

Why not just…?

Why not just use finer fixed bins over range, elevation, incidence and class? This is the most serious threat, and the fixed-grid ablation exists to settle it. If a tuned fixed grid matches the tree, the honest conclusion is that the contribution is the choice of conditioning variables and the switch to surface-density normalization, not the adaptive partition. We would report it that way. Our prediction rests on counts. A four-to-five-dimensional grid fine enough to resolve near-range incidence effects will leave far-range rare-class cells with a handful of real points. The adaptive-grid result of Differentially Private Grids for Geospatial Data suggests that is exactly where fixed resolution loses.

Why not just estimate density ratios point-wise with nearest neighbors in the condition space, with no partition at all? A continuous ratio estimator avoids cell boundaries but has the same variance problem in sparse regions, and it adapts poorly because its bandwidth is global. We will include a $k$-nearest-neighbor ratio estimator with tuned $k$ as an extra baseline. If it ties, the tree's remaining advantages are interpretability (readable rules such as "glass beyond 30 m: keep 5%") and constant-time lookup, and we would say so.

Why not just learn the dropout end-to-end, as ASM does? Learned point-wise masking against a discriminator gave 22.8% on SemanticKITTI in ASM, far below DGT's non-learned 43.1% in DGT-ST. The DGT-ST code notes that its own adversarial stage is unstable. The evidence so far favors statistical translators, so we compare against ASM's reported numbers rather than building on it.

Why not just resample synthetic points onto the target sensor's beam elevations? This reproduces beam geometry, and it is a strong, cheap baseline. We found no reported result for it on Lacuna. It cannot produce material- or angle-dependent losses, though. We will run it alone and composed with our tree. If beam resampling plus DGT rings already closes the gap, the incidence and class axes are not earning their place.

Proposed Experiments

We train on SynLiDAR and evaluate on SemanticKITTI (64-beam), SemanticPOSS (40-beam), and the nuScenes split (32-beam), using the protocols of DGT-ST and ASM. We report mIoU over three seeds, per-class IoU, and IoU binned by range and by incidence angle.

The decisive check is where the gains appear. The hypothesis predicts gains over DGT concentrated in thin and reflective classes and at grazing incidence, with near-parity on roads and buildings. A uniform small gain would point to a regularization effect rather than better density matching. As a model-free diagnostic, a PQMass chi-squared test in the condition-density space, run on held-out real scans, should show translated synthetic data closer to real under our tree than under rings or a fixed grid. If that distributional distance improves but mIoU does not, density matching has saturated as a lever, which is itself worth knowing.

Two further checks probe robustness. The class-agnostic variant measures how much the method depends on pseudo-label quality. Training on SemanticKITTI-fitted trees and testing on nuScenes probes whether the learned rules transfer across sensors or must be refit per target.