Measuring Real LiDAR Corruption with an L0 Fit, Then Applying It to Synthetic Scans

Self-driving perception models trained on simulated laser scans (LiDAR) break down on real sensors. A large part of the reason is that real scans are damaged in ways simulators do not reproduce. Points go missing on glass, dark paint and distant surfaces, and some returns that do arrive are wrong, such as mixed readings at object edges or clutter from rain and multipath. We propose to measure this damage directly on unlabeled real scans and then apply it to clean synthetic ones. The measurement uses an L0 data-fidelity fit, an objective that counts how many measurements disagree with a reconstruction rather than how badly they disagree. That property lets the fit treat wildly wrong returns as discarded outliers instead of averaging them in. The fit recovers, for each real scan, an estimate of the clean underlying geometry together with an explicit map of which entries were dropped or corrupted. These pairs supervise a corruption model, which is then run on synthetic scans before training a segmentation network.

The current approach to this problem is adversarial. Adversarially Masking Synthetic to Mimic Real (ASM) learns to drop synthetic points until a domain discriminator can no longer tell them apart from real ones. It reaches 22.8% mIoU on SynLiDAR→SemanticKITTI, against 15.2% with no adaptation. Its masks, however, come from a local, shallow-layer generator that never sees what real corruption actually looked like. It only sees whether a critic was fooled. Real dropout is spatially organized. It runs along whole laser rows, across entire building facades, and it falls off with distance, so modeling it needs long-range context. Our bet is that observed real corruption, recovered by an outlier-proof fit and modeled with row-and-column attention, is a better training signal than an adversary.

Core Claim

The claim has three parts. First, a range image is the 2D grid view of a LiDAR scan, with laser rows by azimuth columns. Sim-to-real sensor damage on a range image has the same structure as impulse noise on a photograph: a sparse-to-dense set of entries replaced by values unrelated to the truth. Second, the tool built for impulse noise, L0 fidelity with an explicit binary "this entry is clean" variable, can recover real corruption maps without labels. Third, both the prior used inside that fit and the corruption model trained afterwards need a non-local receptive field aligned with the sensor's rows and columns. This holds because real holes are too large and too structured for local operators.

The method's source is ℓ0TV. It lifts the NP-hard L0 problem into a continuous program with an auxiliary binary mask identifying clean pixels, and it restores images with up to 90% impulse noise where L1-TV, median filtering and Adaptive Outlier Pursuit fail. Every L0 restoration method in the file regularizes with local gradient operators: TV in ℓ0TV, and overlapping-group or fractional-order TV in Boosting MR Image Impulse Noise Removal and MR Image Restoration with FOTV and Recursive Filtering. None of them is applied under domain shift. Those MR papers also constrain the design: they report strong sensitivity to the regularization weight and to the fractional order (around 1.9 for Gaussian blur versus around 1.3 for average blur). We therefore replace hand-tuned regularizers with a learned prior rather than adding more of them.

The non-local operator comes from CCNet. Its criss-cross attention lets each position attend only to its own row and column, and two recurrent passes reach the whole grid. It uses 11× less GPU memory and 85% fewer FLOPs than a dense non-local block. On a range image, rows are laser beams and columns are azimuth directions, which are exactly the axes along which dropout is correlated. The operator is unusually well matched to the data.

The proposal must also address a warning. On the Role of Receptive Field in Unsupervised Sim-to-Real Image Translation shows that a translation critic with a large receptive field rewrites synthetic geometry to match real-domain shapes, which makes the synthetic labels invalid. Shrinking the critic preserves shape but weakens realism. We avoid this trade-off differently. Our corruption model has a global receptive field, but its output space can only delete a synthetic point or perturb its range locally. It cannot move objects or change their shape. Global context is used to decide where damage occurs, never to change what the scene contains.

Research Program

The work proceeds in three stages.

Stage 1: a completion prior trained on synthetic scans. We train a range-image completion network $G$ on SynLiDAR range images, where clean geometry is known. Inputs are corrupted with generic random holes and spurious values. The network uses criss-cross attention blocks over beam rows and azimuth columns. Training on synthetic scans means $G$ inherits some geometric sim-to-real gap. SynLiDAR treats geometric appearance and sensor sparsity as separable translation problems, which suggests this gap is secondary to the sensor gap. We test that assumption below.

Stage 2: the L0 fit on each unlabeled real scan. For a real range image $y$, let $\Omega$ be the set of entries with a return. We solve

$$\min_{x,\,z}\; \big\|P_\Omega\big(y - x\big)\big\|_0 \quad \text{subject to}\quad x = G(y, z),$$

where $x$ is the estimated clean range image, $P_\Omega$ keeps only entries that returned a measurement, and $z$ is the binary keep mask that $G$ conditions on. We alternate two steps. In one, the mask is updated by hard thresholding the residual $|y_i - x_i|$ with a tolerance tied to sensor range precision; hard thresholding is the proximal step of the L0 norm. In the other, $G$ re-completes the scan given the current mask. The fit outputs the completed geometry $\hat{x}$ and two corruption maps: dropouts $D$, the entries outside $\Omega$, and spurious returns $S$, the entries in $\Omega$ that the fit rejected, each with a residual offset.

Stage 3: a corruption model trained on real pairs. A corruption model $C$ is trained to predict, from geometry $\hat{x}$ alone (range, local surface normal, height), the probability of dropout and of a spurious return at each entry, plus a displacement distribution for spurious returns. It is trained by supervised likelihood on real $(\hat{x}, D, S)$ triples, again with row/column attention. At training time for segmentation, we sample corruptions from $C$ on clean synthetic scans and train the same segmentation backbone ASM uses. Standard pseudo-label self-training can optionally be added on top, but it is reported separately.

Stage 3 also yields an intermediate scientific product: a held-out test of how predictable real LiDAR corruption is from geometry. We can measure this directly by asking $C$ to predict observed dropouts on real scans it never saw.

Baselines

The naive reference is source-only training on clean SynLiDAR. ASM reports it at 15.2% mIoU on SemanticKITTI and 11.5% on nuScenes. We also include uniform random dropout, matched to the real scans' average dropout rate, as the cheapest form of noise injection.

The strongest existing method is ASM Adversarially Masking Synthetic to Mimic Real. It reports 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes, ahead of CoSMix and SqueezeSegV2, which are both at 20.0% on SemanticKITTI. It is the right competitor because it attacks the same mechanism, synthetic points altered to resemble real sensor damage, using the opposite supervision signal (an adversary rather than observed real damage). We will quote its reported numbers and also rerun it in our codebase, with the same backbone, the same unlabeled real scans, the same number of training iterations and the same hyperparameter-search budget, so that any gain is not a reimplementation artifact. Beating it means a mean improvement over three seeds that exceeds the seed spread on both target datasets. We would treat roughly two mIoU points as a meaningful margin, given that ASM's own margin over CoSMix is 2.8.

We also report the PCT translator from SynLiDAR, which learns dropout via 2D range-image translation and reports a 3.5-point UDA gain. The file does not give its absolute SynLiDAR→SemanticKITTI mIoU, so we will rerun it under our protocol rather than quote a number.

Mask transplant copies observed real dropout masks onto synthetic range images with no modeling. It is the most dangerous simple competitor and is treated as decisive (see below).

Three ablations isolate the proposal's ingredients. The first replaces the learned non-local prior in Stage 2 with ℓ0TV's local TV prior, which isolates the receptive-field argument for completion. The second replaces criss-cross attention in $C$ with a convolutional model of equal parameter count, which isolates it for corruption modeling. The third replaces the L0 fit with a masked L1 fit that never rejects observed returns, so no spurious-return map is produced. This isolates the outlier-rejection ingredient.

Why not just…?

Why not just transplant real dropout masks onto synthetic scans? Copying a real scan's missing-return pattern onto a synthetic range image costs nothing. It reproduces row structure and distance falloff exactly, and it needs no completion, no L0 fit and no model. Its weakness is that real holes follow real objects: a glass facade in the real scan becomes a hole on a synthetic road. This breaks the correlation between corruption and semantic class that a segmenter must learn to tolerate, such as missing car windows and missing building glass. We do not know from the file how much that correlation matters, so this is a real threat. If transplant comes within about one mIoU of our method on both targets, the conclusion is that geometry-conditioned corruption is not worth modeling, and the paper becomes a negative result about ASM-style conditioning as well.

Why not just use masked L1 or Huber fidelity instead of L0? Missing returns are already flagged in the range image, so for pure dropout the L0 fit reduces to fitting only the observed entries. Here L0 adds nothing, and a reviewer is right to say so. The L0 term earns its place only through the second corruption type: returns that arrive but are wrong. Under L1, such returns pull the completion toward the bad value. ℓ0TV shows this failure becomes severe at high corruption density, where L1 methods could not recover structure that L0 recovered. The masked-L1 ablation settles this question. If it matches the L0 fit downstream, the spurious-return channel is unimportant on these sensors and we will report that.

Why not just add criss-cross attention to ASM's mask generator? This would test whether long-range context alone, without real supervision, is enough. We run it as an extra baseline. Our reason to expect it to fall short is that ASM's discriminator only needs dropout statistics to match. Pattern details that a critic cannot detect, such as whether the dropped points are on the correct class, are never rewarded. The receptive-field paper shows a second risk: a global adversarial signal tends to exploit whatever global cue separates the domains.

Why not just label some real scans? SynLiDAR reports that translated synthetic data matched SemanticPOSS performance with 40% fewer real labels, so synthetic data saves labeling effort but does not replace it. Our setting is unsupervised adaptation, which is complementary to labeling. A 10%-labeled fine-tuning arm will show whether the gains persist once some real labels exist.

Proposed Experiments and Decisive Checks

The main experiments are SynLiDAR→SemanticKITTI and SynLiDAR→nuScenes, evaluated with per-class and mean IoU on the standard validation splits. The supervised oracle of 55.5% on SemanticKITTI, reported in ASM, marks the ceiling. We do not expect sensor modeling alone to close a gap that large. The goal is to show that observed corruption beats adversarially invented corruption.

Before any segmentation training, two checks test whether the pipeline measures real corruption at all. The first is a self-consistency test on real data. We hide a random subset of observed real returns, run the L0 fit, and measure range error on the hidden entries. We also inject synthetic spurious returns and measure detection precision and recall. This compares the learned non-local prior with the local TV prior, with the gap broken down by hole size. If the receptive-field argument is right, the advantage should grow with hole extent. The second is a prediction test. On held-out real scans we score $C$'s dropout probabilities against the observed dropout map using area under the ROC curve, comparing $C$, its convolutional ablation, and ASM's mask generator queried on completed real geometry.

The decisive comparison is against mask transplant and ASM. The proposal succeeds only if it beats both on both target sensors. The ablations must then show which ingredient carries the gain: completion receptive field, corruption-model receptive field, or L0 rather than L1 rejection.

Two secondary measurements test our account of why the method works. First, we retrain a small domain discriminator on segmenter features. If our corruptions remove the "clean versus damaged" shortcut more thoroughly than ASM's, its accuracy should be lower. Second, we measure sensitivity of mIoU to the residual threshold in the L0 fit. This addresses the hyperparameter fragility documented in the MR restoration work.