Models that label LiDAR point clouds after training only on simulated scans perform poorly on real sensors. Current fixes work offline and target one sensor at a time. Each one learns, by adversarial training, to make synthetic scans look like scans from one particular real sensor: matching its noise, its point dropouts, or its beam pattern. If you switch to a different sensor, you retrain the whole pipeline. We propose a different approach. A single segmentation network is trained in simulation against a broad randomized family of virtual sensors. At deployment it reads a small unordered set of unlabeled scans from the real sensor and adapts to that sensor in one forward pass, with no gradient updates. The scans form a set because their order carries no information, so a permutation-invariant encoder can summarize them into a "sensor code" that conditions the segmenter. A permutation-invariant encoder is a network whose output does not change when its inputs are reordered.
The question is narrower than "does context help." Domain randomization alone (training on many virtual sensors with no context) may already give a model that is robust to any sensor. The network can also often infer the sensor from the very scan it is labeling. Context only helps if some property of the sensor, such as dropout on glass or wet surfaces, intensity calibration, or mounting geometry, cannot be identified from one scan but can be from several. The work is designed so this question gets a clear yes or no answer on public sim-to-real LiDAR benchmarks.
Core Claim
A segmenter conditioned on a learned set summary of $n$ unlabeled target scans should beat three alternatives on unseen real sensors: the same segmenter trained with identical sensor randomization but no context, test-time re-estimation of normalization statistics, and per-sensor adversarial adaptation. The gain should grow with $n$ beyond $n=1$. If the gain is flat in $n$, the claim is false, because it would mean the scan being labeled already reveals everything about the sensor.
Where This Sits
The mechanism comes from Towards Context-Aware Domain Generalization. That paper encodes an unordered set of unlabeled samples from an environment into a summary vector and feeds it to the predictor next to the input. It also states when this can help: the context must carry information about the label beyond the input, the environment must not be identifiable from a single sample, and the input–label relationship must vary across environments. It tested these ideas on ColoredMNIST, a Simpson's-paradox toy problem, and BikeSharing. It never tested them where each context element is itself a large point set, or where the environments are sensors. Its second condition, that one sample does not reveal the environment, is exactly what is uncertain for LiDAR, and our experiments test it directly.
The problem setting and the competitors come from the point-cloud sim-to-real literature. Three papers each build a fixed translation toward one known target sensor:
- SynLiDAR splits the gap into appearance, handled by a GAN, and sparsity, handled by translating range images.
- Adversarial Spatial Masking (ASM) learns point-drop masks so that clean synthetic scans stop being trivially distinguishable from real ones.
- SIRA-PCR shifts points within the convex hull of their neighbors to imitate real sensor noise while keeping the geometry intact.
We reuse their corruption ideas as components of the randomized sensor family. The difference is that we condition the model at inference time instead of retraining a translation per sensor.
Two papers constrain the encoder design. Transformers Generalize DeepSets shows that attention generalizes static sum or mean pooling. That motivates attention, but it also makes mean pooling a mandatory ablation. Statistical Foundations of PFNs shows that softmax attention over large contexts has a bias that stops shrinking as the context grows. We therefore keep contexts small (up to 64 scans) and look for this plateau rather than assume large $n$ is free.
Research Program
Simulating many sensors. We build a virtual-sensor generator on top of SynLiDAR. Each synthetic sweep is projected into a range image and resampled under a sampled sensor description:
- beam count and vertical angles,
- mounting height and pitch,
- range-dependent noise,
- material- and incidence-dependent dropout,
- intensity response.
ASM-style learned drop masks and SIRA-style hull-constrained point shifts serve as extra corruption families. One honest limitation is that resampling can only remove or perturb points, never add detail. Sensors denser than SynLiDAR are therefore out of scope.
Model. A shared point encoder compresses each context scan into a few tokens. A transformer without positional encodings attends across the tokens of all $n$ scans and pools them into a sensor code $z$. The code modulates every block of a standard sparse-convolution segmenter through feature-wise scale and shift. Training samples one virtual sensor per episode, draws $n \in [1, 64]$ context scans and a separate query scan from that sensor, and minimizes segmentation loss on the query. The context scans are never labeled, so deployment needs nothing but raw scans from the new sensor.
Analysis. On held-out virtual sensors, where the true sensor parameters are known, we fit linear probes from $z$ to each parameter. This shows which sensor properties the code actually captures. It also tells us whether the probe accuracy for a property, such as dropout rate, improves with $n$ while the probe accuracy for beam count saturates at $n=1$.
Baselines
Source-only reference. This is the naive reference point: the backbone trained on SynLiDAR as-is. ASM reports 15.2% mean intersection-over-union (mIoU) on SemanticKITTI and 11.5% on nuScenes for this setting. We reproduce it with our backbone.
Context-free sensor randomization. This is the key ablation. It uses the same generator, backbone, data, and compute, but $z$ is replaced by a learned constant. It isolates what reading the context adds beyond robustness to randomization. We give it the same hyperparameter search budget as the full model.
Test-time normalization re-estimation. This baseline recomputes the backbone's normalization statistics on the same $n$ unlabeled target scans. It is the strongest cheap competitor, because it is itself a mean-pooled set statistic over target data. We apply it both to the context-free model and on top of ours.
Mean-pooled set encoder. This ablation is identical to the full model except that a DeepSets-style mean replaces attention.
ASM. This is the strongest existing method. It is trained separately for each target sensor using that sensor's unlabeled data. ASM reports 22.8% mIoU on SynLiDAR→SemanticKITTI and 17.0% on SynLiDAR→nuScenes ASM. On SemanticKITTI the same paper reports CoSMix at 20.0% and a fully supervised real-data oracle at 55.5%. We report the published ASM numbers, and we also rerun ASM with our backbone and the same total unlabeled target data, so that differences in backbone do not decide the comparison.
What counts as winning. One model, never trained on target data, must match or exceed per-target ASM on both targets. It must also beat the context-free and normalization baselines by a margin larger than the spread across three random seeds.
Why not just…?
Why not just randomize sensors broadly and skip the context? This is the most serious threat. If the segmenter can infer the sensor from the scan it is labeling, context-free randomization will match our method. Neural Posterior Domain Randomization and CAPTURE argue that broad randomization yields overly conservative behavior in robot control. That evidence is about dynamics, not perception, so it does not settle the question here. The context-free ablation and the curve of mIoU against $n$ settle it directly. If the curve is flat, we will report that context is redundant for LiDAR.
Why not just re-estimate normalization statistics on target scans? This method already uses unlabeled target sets and costs almost nothing. It can only shift first and second moments of the features. It cannot, for example, change how the model treats a missing return on a reflective car window. We expect our method to win on classes affected by dropout, such as cars, windows, and wet road. If the overall gap is small, we will report the per-class breakdown.
Why not just retrain ASM or SynLiDAR's translator for each new sensor? This gives the best number for any single known sensor. Its cost is a full adversarial training run per sensor and per hardware change. Our claim is parity at zero retraining, not superiority on the sensor ASM was trained for. A held-out third target makes the comparison fair: SemanticPOSS, evaluated on classes shared with SynLiDAR. We found no reported ASM result on SemanticPOSS on Lacuna.
Why not just mean-pool the scan summaries? It may suffice. Following Transformers Generalize DeepSets, attention should matter when the relevant evidence is spread across scans, such as dropout that appears only at certain incidence angles. The ablation decides this. If pooling ties attention, we will adopt pooling.
Decisive Checks
Five checks determine the outcome. The first two decide whether the core claim holds:
- mIoU as a function of context size. We plot mIoU against $n \in \{1, 4, 16, 64\}$ on SemanticKITTI, nuScenes, and SemanticPOSS. A rise beyond $n=1$ that exceeds the context-free model supports the claim. A flat curve falsifies it.
- Mismatched context. Feeding nuScenes context to a model labeling SemanticKITTI scans should drop accuracy below that of matched context. If it does not, the model is ignoring $z$.
The remaining three checks ground or bound the result:
- Probes on held-out virtual sensors. These show which sensor parameters $z$ recovers, which ones need more than one scan, and whether recovery plateaus at large $n$, as the PFN localization analysis predicts.
- Permutation invariance. Shuffling the order of context scans must leave outputs unchanged to numerical precision. This is a sanity check on the implementation, not a result.
- Closing the gap to real labels. We report how much of the distance between source-only (15.2%) and the oracle (55.5%) on SemanticKITTI the method closes, against ASM's 22.8%.
References
- Towards Context-Aware Domain Generalization: Representing Environments with Permutation-Invariant Networks
- SynLiDAR: Learning From Synthetic LiDAR Sequential Point Cloud for Semantic Segmentation
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- SIRA-PCR: Sim-to-Real Adaptation for 3D Point Cloud Registration
- Transformers Generalize DeepSets and Can be Extended to Graphs & Hypergraphs
- Statistical Foundations of Prior-Data Fitted Networks
- Neural Posterior Domain Randomization
- Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identifications