Robots and cars localize against prior maps. The most compact maps today throw away raw geometry and keep only labeled landmarks. One system stores each LiDAR scan as a set of object centers tagged with a class, $(x, y, z, c)$, which shrinks a 1.41 MB scan to 1.33 kB and still localizes to about 0.1 m on KITTI (That's My Point). These maps inherit every mistake of the segmentation network that produced them. If a pole is labeled vegetation, or two parked cars merge into one cluster, the stored landmark is wrong in a discrete way. The robust cost functions and RANSAC that registration relies on assume errors are small and scattered, so they cannot remove errors like these (VSO, SemaGlue).
We propose storing each map tile as a short latent code instead. A shared hypernetwork (a network that outputs the weights of another network) decodes the code into a tiny coordinate network, or neural field: for any 3D point, it returns a distance to nearby static structure and a probability distribution over classes. The code is trained so that the decoded field produces correct poses when a registration solver uses it, not so that it reconstructs the scan. A generative model over codes, conditioned on the noisy segmented scans used to build the map, draws several plausible versions of each tile. Where these versions disagree, the segmentation evidence was ambiguous, and the localizer weights those regions down. The bet is that, at the same number of bytes, a handful of sampled field hypotheses localizes more reliably under realistic segmentation errors than hard-labeled or soft-labeled landmark constellations.
Core Claim
The claim has three separable parts, and the experiments test each one. The first is about what to compress. A map is good if it lets you recover pose, so the compression objective should be registration behavior. This carries the behavioral-loss lesson of Structure Is Not Enough from classifiers into geometry. That paper found that weight reconstruction trained only on numeric error left SVHN accuracy at 74.5%, while adding a loss on network outputs restored it to 90.4%. The second is about what form the map takes. A field that accepts the live segmenter's soft class probabilities and returns a continuous cost encodes surfaces as well as centers. Surfaces should help where centroid constellations are sparse or shifted by occlusion. The third is about why the model is generative rather than deterministic. Segmentation errors made while building the map are multimodal: the object either is or is not a car. A deterministic encoder averages the modes. A sampler keeps them as distinct hypotheses.
Relation to Prior Work
The closest work is Encoding Semantic Priors into the Weights of Implicit Neural Representation (SPW). It generates the weights of a coordinate network from frozen EfficientNet features for 2D images and medical volumes, gaining up to 1.4 dB PSNR on Kodak. We differ in three ways. SPW is deterministic, its objective is reconstruction fidelity, and it simply inherits any error made by its upstream feature extractor. Our generator samples, our objective is pose recovery, and our training deliberately exposes the model to segmentation errors.
HyperDiffusion and inr2vec supply the method family: diffusion over the weights of fitted shape networks, and encoders that map those weights to compact codes. Both work on isolated CAD objects and both depend on shared random initialization to keep weights comparable across models. We sidestep that alignment problem. The decoder is a shared hypernetwork, so every tile's weights live in one coordinate system by construction, and nothing has to be matched across a model zoo. This answers the scene-level form of the permutation problem that DeepWeightFlow handles with Git Re-Basin and PCA for classifiers. We reuse DeepWeightFlow's flow-matching sampler, a generator that learns a straight path from noise to data, but run it over codes rather than raw weights.
SNI-SLAM shows that a semantic neural field can down-weight wrong 2D labels through cross-attention. It optimizes 6.2M parameters per scene online, maps at about 2.5 FPS, and does not compress. It motivates our field form but is not a compact map.
Two papers constrain the design. The "weak link" finding in Hyper-Representations for Pre-Training and Transfer Learning says that one badly reconstructed layer can break a decoded network. This favors a hypernetwork that outputs only a low-rank modulation of a shared base field, rather than full per-tile weights. Meynent et al. also observed that behavioral probes drawn from the wrong distribution hurt. Our probes are therefore real scans with realistic errors, not noise.
Research Program
A tile is a fixed-size piece of the map, for example 40 m on a side around a keyframe. It is described by a code $z$ of a few hundred bytes. A shared hypernetwork $H$ turns $z$ into weights for a small field $f_{H(z)}$. The field maps a point $p$ and a class distribution $q$ from the live segmenter to a registration cost. Internally, the cost combines a predicted distance to static structure with the agreement between $q$ and the field's own class distribution at $p$.
For training we take KITTI and KITTI-360 sequences with ground-truth poses. Each training tile is built from multiple passes of segmented scans. During training we corrupt the segmentations by sampling class swaps from the segmenter's measured confusion matrix, merging and splitting clusters, and masking regions to mimic occlusion. The objective is
$$ \mathcal{L}(z) = \mathbb{E}_{S, T_0}\big[\, \| \hat{T}(f_{H(z)}, S, T_0) \ominus T^\star \| \,\big] + \lambda \, \mathbb{E}_{p}\big[\, (d_{H(z)}(p) - d^\star(p))^2 \,\big]. $$
Here $S$ is a held-out query scan carrying corrupted soft labels. $T_0$ is a perturbed initial pose. $\hat{T}$ is the pose returned by a few unrolled, differentiable Gauss–Newton steps on the field's cost. $T^\star$ is the true pose, and $\ominus$ is the pose difference. $d$ and $d^\star$ are the predicted and true distances to static structure, and $\lambda$ is a weight. The second term keeps the field geometrically sensible away from the solver's basin. The first term is the behavioral loss.
The generative stage is a conditional flow-matching model over $z$. Its condition is a point-set encoding of the noisy segmented scans available when the map is built. When building a map, we draw $K$ codes per tile and store all of them, so the total byte budget includes all $K$. At query time the localizer runs the solver against each sampled field. It combines the resulting costs with a robust mixture, so that points where the samples disagree count for less.
We also build a variant whose codes come from an inr2vec-style encoder over separately fitted per-tile fields. This tests whether the weight-space encoding route adds anything over encoding the observations directly. We expect it may not, and we will report either outcome.
Baselines
The naive reference is the hard-labeled centroid map of That's My Point, with its learned graph matcher, weighted SVD and RANSAC. We run it unmodified, using Cylinder3D segmentations as the original does. It is also the strongest existing method for compact LiDAR localization in this file. The paper reports 0.1 m translation and 0.5° rotation error on KITTI at 1.33 kB per scan, with over 94% recall within 0.5 m/5° across a two-year gap between KITTI and KITTI-360. No reported result was found on Lacuna for that method under controlled segmentation corruption, so we will produce that curve ourselves. To beat it, our method must match its clean recall at no more than 1.33 kB per tile, and keep a statistically significantly higher recall at every corruption level beyond the segmenter's native error rate.
The soft-label centroid baseline gives the same pipeline a class histogram per centroid in place of the argmax label. It is quantized so that bytes match ours. It is retrained with the same corruption augmentation and the same tuning budget. This is the competitor most likely to erase our advantage, because it directly answers the hard-assignment problem.
The deterministic ablation uses our architecture with the flow model replaced by a regression encoder. It gets one larger code at the same total bytes as $K$ sampled codes. It isolates whether sampled hypotheses beat added capacity.
The reconstruction-trained ablation drops the pose term from the loss. It isolates the behavioral objective. All learned methods share training sequences, corruption models, GPU hours and hyperparameter search size, and all are evaluated on held-out sequences.
For context we also report a PCA-and-quantization compression of per-tile fields, in the spirit of DeepWeightFlow's linear compression. We additionally report dense point-to-plane registration against raw scans as an accuracy ceiling at 1000x the storage.
Why not just…?
Why not just store soft class probabilities with each centroid? This is the real threat. If soft-label centroids match our corrupted-segmentation curves, the field and generative machinery are unjustified, and we will say so. Our reason to expect a gap is that many segmentation failures are geometric, not categorical. Merged clusters, occlusion-shifted centers and missed objects move or delete the landmark itself, and a class histogram cannot repair that (That's My Point notes exactly this sensitivity). We run cluster corruptions separately from label corruptions so the result says which error type each method handles.
Why not just fuse labels over repeated passes with a Bayesian filter? Filtering over class probabilities works when errors are independent across passes. VSO reports up to 45% less drift with an EM treatment of class distributions. The same work notes that systematic errors, such as a parked car mistaken for static structure, pull the cost field the wrong way. Repeated passes with the same segmenter repeat the same systematic errors. We give all baselines the same multi-pass fusion and test with errors that are correlated across passes.
Why not just fit a neural field online, as in NeRF-SLAM? SNI-SLAM shows this is accurate indoors. It uses millions of parameters per scene and maps at about 2.5 FPS. It is a dense map, not a kilobyte map, so it belongs on the accuracy-ceiling side of the comparison rather than the compact side.
Why not just learn a latent per tile with a shared decoder and skip weight-space generation? That is in fact what our main model does: the hypernetwork is a shared decoder, and the weight-space encoder variant is only an ablation. The new elements are the registration objective and the sampled hypotheses. Weight-space modeling of zoos is not one of them.
Proposed Experiments
The decisive experiment is a robustness curve on KITTI and the KITTI-360 cross-session split. The x-axis is the severity of segmentation corruption. It starts from the native segmenter output; for reference, KITTI-360 reports 81.2% mIoU for static-object segmentation (KITTI-360). Severity increases through confusion-matrix label swaps, cluster merges and splits, and occlusion masking. A second axis swaps in a weaker segmenter rather than synthetic noise. The y-axis is recall within 0.5 m/5°, alongside median translation and rotation error. Every method sits at a matched budget of 1.33 kB per tile, and we sweep budgets from 0.3 to 3 kB.
The generative claim holds only if $K$ sampled codes beat one deterministic code of equal bytes, and only if sample disagreement correlates with injected corruption locations. We check the correlation with the area under the precision–recall curve for detecting corrupted regions from disagreement. The behavioral claim holds only if the pose-trained model beats the reconstruction-trained one at equal bytes.
A secondary test targets longitudinal ambiguity on long, repetitive KITTI stretches. Ma et al. found that lane polylines leave 1.12 m median longitudinal error against 0.05 m lateral (Exploiting Sparse Semantic HD Maps). We compare error along the direction of travel for fields and centroids on segments we select for sparse landmarks.
We also report the cost of decoding and solving per tile against the roughly 20 ms registration time of the centroid method. The goal is to stay within a small multiple of it.
The result that would end the project is soft-label centroids, trained with the same augmentation, matching our curve within noise. In that case the conclusion is that hard argmax was the whole problem and the field is unnecessary.
References
- That's My Point: Compact Object-centric LiDAR Pose Estimation for Large-scale Outdoor Localisation
- VSO: Visual Semantic Odometry
- Matching While Perceiving: Enhance Image Feature Matching with Applicable Semantic Amalgamation
- Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction
- Encoding Semantic Priors into the Weights of Implicit Neural Representation
- HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion
- Deep Learning on Implicit Neural Representations of Shapes
- DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights
- SNI-SLAM: Semantic Neural Implicit SLAM
- Hyper-Representation for Pre-Training and Transfer Learning
- KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D
- Exploiting Sparse Semantic HD Maps for Self-Driving Vehicle Localization