Simulated LiDAR scans are too clean. A simulator casts perfect rays onto perfect surfaces, while a real sensor loses points on dark, shiny, or distant surfaces. One recent fix trains a small network to delete points from synthetic scans until a discriminator can no longer tell them from real ones. Deleting a point is a yes/no decision, and gradients cannot flow through yes/no decisions directly, so the method uses a standard gradient approximation that becomes very noisy when the decisions are forced to be crisp. We propose replacing that approximation with a lower-noise version built on conditional averaging (Rao-Blackwellization). For independent per-point binary decisions, this version can be computed exactly at almost no cost. We then test whether lower gradient noise actually yields better learned dropout and better real-world segmentation.
The question is narrow, and that is intentional. The file documents a large gap between the best learned-dropout method (22.8% mean intersection-over-union, mIoU, on real SemanticKITTI) and a real-data oracle (55.5%). It also documents that the gradient estimator inside that method is known to misbehave in exactly the regime the method needs. What nobody has measured is how much of the remaining gap comes from the optimizer and how much from the problem itself. This proposal is built to separate the two, and it treats a null result as informative.
Core Claim
Adversarially Masking Synthetic to Mimic Real (ASM) inserts a module that predicts a drop probability for each synthetic LiDAR point. It samples binary keep/drop masks and trains the module adversarially against a domain discriminator. It does this with the Straight-Through Gumbel-Softmax estimator: the forward pass uses a hard 0/1 sample, and the backward pass pretends the sample was a smooth "softmax at temperature τ" of the noisy logits. Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator shows two things about this estimator. First, much of its variance comes from the random noise and not from the decision itself, because infinitely many noise draws produce the same 0/1 outcome. Second, averaging the backward gradient over the noise conditioned on the realized decision removes that variance. Their Gumbel-Rao estimator does this with K Monte Carlo draws and evaluates the network only once. At τ = 0.01 on ListOps it reached about 67% accuracy against about 39% for the plain estimator.
Our claim has two parts. The first is technical. For binary decisions, the Gumbel-Rao conditional expectation needs no Monte Carlo at all. Write the point's logit as $a$ and the logistic noise (the difference of two Gumbels) as $L$. The surrogate gradient is then
$$ g(a,\tau,D) = \mathbb{E}\left[\tfrac{1}{\tau}\,\sigma'\!\left(\tfrac{a+L}{\tau}\right) \,\middle|\, D\right], \qquad D = \mathbb{1}[a+L>0], $$
where $\sigma'$ is the derivative of the logistic sigmoid. This is a one-dimensional integral of a smooth function over a truncated logistic distribution. It can be precomputed as a table in $(a,\tau,D)$ or evaluated with a few quadrature nodes. This gives the K→∞ limit of Gumbel-Rao for millions of points at roughly the cost of the plain estimator.
The second part is empirical. With this gradient, the masking module can be trained at low temperatures, where its masks are genuinely binary, without destabilizing the adversarial game. We predict two effects. The learned dropout patterns should match real scans more closely. Transfer should improve on SynLiDAR→SemanticKITTI and SynLiDAR→nuScenes in a way that is stable across seeds.
We do not claim the estimator removes bias. Conditioning reduces the variance of a biased straight-through estimator but does not touch its bias, as the Gumbel-Rao paper itself notes. Our experiments therefore include an unbiased estimator as a check.
Research Program
The first deliverable is the estimator. It is a drop-in autograd function for binary straight-through masks, backed by a precomputed table of $g$. We will verify it against GR-MCK with large K and measure its overhead relative to plain straight-through. Because the segmentation backbone sees only the hard mask, no extra backbone passes are needed. This is the same structural property that makes Gumbel-Rao and Orbit Diffusion cheap: both average inside the backward pass while running the network once. Orbit Diffusion supplies the general lesson that augmentation-like randomness is best averaged where the network does not see it. It does not supply the method, since it handles group symmetries on molecular coordinates rather than discrete sensor corruptions.
The second deliverable is a reimplementation of ASM on its two benchmarks. We will verify that our reproduction lands near the reported numbers before changing anything.
The third deliverable is a diagnostic study, which is the actual scientific contribution. It measures four quantities. The first is gradient variance of the masking-module parameters under each estimator, estimated from repeated backward passes at fixed parameters. The second is stability of the min-max game, measured by discriminator accuracy oscillation and seed-to-seed spread. The third is how closely learned masks match real dropout, measured by comparing per-beam and per-range dropout rates of masked synthetic scans with those of real unlabeled scans. The fourth is downstream mIoU. All four are measured across a temperature sweep. The hypothesis predicts a specific pattern. At high τ the estimators should agree. As τ falls, plain straight-through should degrade sharply while the conditioned estimator holds. If instead mIoU is flat across estimators even where variance differs substantially, then estimator noise is not what limits ASM, and the 22.8% to 55.5% gap is semantic or geometric. That would be a useful negative result for the line of work that SynLiDAR opened by splitting the gap into appearance and sparsity.
Baselines
The naive reference point is source-only training on SynLiDAR with no masking. The file reports 15.2% mIoU on SemanticKITTI and 11.5% on nuScenes (ASM). We rerun it with our backbone so that every comparison shares one codebase.
The strongest existing method is ASM itself, with its original straight-through Gumbel-Softmax estimator. The file reports 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes. CoSMix is the strongest other domain adaptation method it lists, at 20.0% on SemanticKITTI. ASM is the right competitor because our change modifies only its gradient estimator. Any gain is therefore attributable to that change, and any failure is attributable to it too. For fairness, ASM gets the same data, backbone, number of training steps, and seeds as our variant. It also gets its own temperature tuning, using the same grid and the same number of runs as our method. This matters because a poorly tuned straight-through baseline would inflate our gain. Beating it means a mean mIoU improvement over at least five seeds that exceeds the seed standard deviation of both methods, on both target datasets, at the best temperature of each method. We would also treat a reduction in seed spread at equal mean as a meaningful win for practitioners. We will report whether our reproduction reaches 22.8%. If it does not, we will compare against our reproduction rather than the reported number.
Two ablations isolate the key ingredient. GR-MCK with small K (1, 4, 16) tests whether the exact conditional expectation matters beyond a cheap approximation. An unbiased REINFORCE-style estimator with a leave-one-out baseline over several masks per scan tests whether straight-through bias, not variance, is the binding constraint.
The non-gradient competitor is mask transfer: copying dropout patterns from real unlabeled range images onto synthetic range images, in the spirit of SynLiDAR's range-image sparsity module. It is described under "Why not just…?" below because it is the most serious threat.
Why not just…?
Why not just raise or anneal the temperature in ASM? At higher τ, straight-through variance is lower but the backward pass describes a soft mask that the forward pass never applies. The bias then grows. The Gumbel-Rao paper's ListOps result shows the low-τ regime is where the plain estimator collapses (about 39% against 67%). Whether this transfers to dropout masks is exactly what our temperature sweep measures. If ASM's best tuned temperature already matches our method, the proposal's practical value is nil, and we will report that.
Why not just draw more Monte Carlo masks per step? Each extra mask costs a full forward and backward pass through a 3D backbone and the discriminator. Paulus et al. note this cost scales linearly in the number of samples, whereas conditioning works inside the backward pass of a single evaluation. We include a matched-wall-clock comparison against ASM with M masks per scan, so the trade-off is measured directly rather than asserted.
Why not just copy real dropout masks instead of learning them? This is a genuine threat. In unsupervised adaptation, real unlabeled scans are available, and their missing-return pattern in range-image coordinates is directly observable. Copying those patterns requires no adversarial game and no gradient estimator. Its weakness is that copied masks ignore content. Real dropout depends on material, incidence angle, and range on this scene's surfaces, and that dependence is what a learned, point-conditioned module can capture. We will run mask transfer under the same backbone and budget. If it matches learned masking, the conclusion is that learning dropout is unnecessary at this level of realism. That would be a substantive finding, and the proposal would pivot to explaining which content-dependent dropout a learned module adds.
Why not just use feature-level alignment and skip input-space masking? ASM reports that feature discriminators exploit the density difference as a shortcut. Similarly, PixelDA found input-level modeling far better than feature alignment on synthetic-to-real LineMod (23.5° against 56.58° for DANN). We therefore do not rerun this comparison and take input-space corruption as the setting.
Proposed Experiments
The main experiments use SynLiDAR as the source and SemanticKITTI (64-beam) and nuScenes (32-beam) as targets, scored by mIoU over at least five seeds. For context we report the 55.5% oracle and the source-only numbers from the file. The estimator check comes first. We confirm that the tabulated binary Gumbel-Rao gradient matches GR-MCK at large K and that its per-step wall-clock is within a few percent of plain straight-through.
The decisive check is the temperature-by-estimator grid, measuring gradient variance, discriminator oscillation, dropout-statistics match to real scans, and mIoU. The hypothesis survives only if two things hold together: variance reduction appears at low τ, and it comes with better dropout matching and better mIoU than the best-tuned ASM. It fails, informatively, in either of two ways. If variance drops but nothing downstream moves, optimizer noise is not ASM's bottleneck. If mask transfer matches every learned variant, learning dropout is not needed at all. The unbiased-estimator ablation then indicates whether any remaining ceiling comes from straight-through bias. That is the natural next target if the variance story turns out to be only half the explanation.
References
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator
- Rao-Blackwell Gradient Estimators for Equivariant Denoising Diffusion
- SynLiDAR: Learning From Synthetic LiDAR Sequential Point Cloud for Semantic Segmentation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks