A robot that maps as it moves must guess what lies behind walls and around corners. Some of the simplest guessers have no neural network at all. Bayesian kernel smoothing spreads observed occupancy and semantic labels into nearby unseen cells. Low-rank tensor completion fills a partially observed grid by assuming it has simple, repetitive structure. Both are fast, need no training data, and give explicit uncertainty. Both also depend on one or two dials that someone sets by hand: a kernel length-scale, or a rank. Those dials directly control how far, and how boldly, the robot hallucinates into unknown space. The authors of S-BKI say so plainly, describing a manually tuned length-scale shared across all classes that "trades off predictive ability and classification accuracy" (Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping).
This proposal argues that the dials are tuned against the wrong target. The usual automatic tuning criteria score how well the model explains cells it has already seen, or they assume missing cells are scattered at random. Examples are marginal likelihood, variational rank pruning, and weight-free min-max formulations. A robot's missing cells are not random. They are contiguous occlusion shadows and regions beyond the frontier, so the quantity that matters is extrapolation into structured holes. We propose choosing hallucination hyperparameters online, separately for each semantic class, by scoring candidate settings on masks that have the geometry of real occlusion. These masks come from two sources. One is ray-cast shadows carved out of the already-observed map. The other is the robot's own later observations of cells it earlier guessed. The bet is that structure-aware selection beats both hand-tuning and observation-fitting criteria on extrapolation accuracy and on navigation cost, and that it closes much of the gap to trained deep hallucinators at a fraction of their latency.
Core Claim
Hyperparameter sensitivity in spatial completion is mostly a mismatch between how hyperparameters are validated and how data goes missing. Tensor completion theory and benchmarks almost always use uniformly random missing entries: the "Suzie" video at 5% sampling, natural images at 99% missing (Completion of High Order Tensor Data with Missing Entries via Tensor-train Decomposition). Under random missingness, a slightly over-smoothed or over-ranked model still has observed neighbors on all sides. Under occlusion, it must reach across a gap of meters with neighbors on one side only. The best length-scale or rank for the first regime should be systematically wrong for the second. Our claim is that per-class hyperparameters selected on occlusion-shaped validation masks transfer across environments of different scale. Hyperparameters fixed offline or selected by observation-fit criteria do not.
Concretely, at time $t$ the robot has an observed cell set $O_t$. Let $M$ be a validation mask drawn from an occlusion-shaped distribution $\mathcal{D}_t$. Let $f_\theta(O_t \setminus M)$ be the completion produced with hyperparameters $\theta$ (per-class length-scales, or a rank tuple) when the cells in $M$ are hidden. We choose
$$\theta_t = \arg\min_{\theta} \; \mathbb{E}_{M \sim \mathcal{D}_t} \sum_{c \in M} w(c)\, \ell\big(f_\theta(O_t \setminus M)_c,\; y_c\big),$$
where $y_c$ is the observed label of cell $c$, $\ell$ is cross-entropy over occupancy and semantic classes, and $w(c)$ increases with the cell's distance from the mask boundary. The weighting rewards true extrapolation rather than edge interpolation. The search over $\theta$ is a small per-class grid or a few golden-section steps, warm-started from $\theta_{t-1}$. The two choices of $\mathcal{D}_t$ are the research question. Self-masking casts virtual rays from past poses to carve realistic shadows into $O_t$ and is available immediately. Delayed verification uses cells the robot hallucinated earlier and has since actually seen, so the mask is exactly the true occlusion distribution, at the cost of lag.
Research Program
We build this selection loop around two training-free completers, because their hyperparameters are explicit and few.
The first completer is S-BKI as published (Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping), with its single length-scale replaced by one length-scale per semantic class. S-BKI provides the kernel inference and variance maps. We change only how its key parameter is set.
The second is a low-rank completer for a local robot-centric tensor. The modes are height, width, and class channel, plus a short stack of recent frames. It uses a Tucker-style sum of nuclear norms with a total-variation term, in the spirit of Tensor Completion Using Bilayer Multimode Low-Rank Prior and Total Variation, and is solved with the accelerated inexact soft-impute of AIS-Impute to keep each frame cheap. Here $\theta$ is the rank tuple, or equivalently the mode weights and threshold. Indoor rectilinear layouts make this plausible: axis-aligned walls and corridors give occupancy matrices low effective rank. Whether that survives clutter is part of what we measure.
The radio-map work shows that low-rank plus smoothness can complete spatial fields from sparse samples (Tensor Completion for Radio Map Reconstruction using Low Rank and Smoothness, DULRTC-RME). That work motivates our second completer. Its sampling pattern is scattered measurements rather than occlusion shadows, though, and it never feeds a planner. We supply the method for the missing piece, tuning under structured missingness, and we close the loop with planning.
Both completers feed a planner that treats hallucinated cells by their predicted probability. We use a cost-weighted A* for point-goal navigation and an information-gain frontier selector for exploration. This lets us measure downstream effects, not just map accuracy. The analysis has three parts. First, we produce sensitivity curves of extrapolation accuracy against $\theta$, per class and per environment, to establish that the optimum moves at all. Second, we measure how well each selection criterion's chosen $\theta$ tracks the oracle per-environment optimum. Third, we compare navigation outcomes.
Baselines
The naive reference point is the optimistic planner, which assumes every unknown cell is free and replans on contact. It runs on the same simulator episodes and the same A and frontier code. Every hallucination method must beat it on path length and replanning count, or hallucination is not paying for itself. Prior work says it should: Planning Paths Through Unknown Space by Imagining What Lies Therein reports frequent dead-end replanning under optimism, and CogniPlan reports a 3.9% navigation travel-length reduction over D Lite. For exploration, classical frontier-based exploration is the matching naive point. UPEN reports 120.2 m² map accuracy and 53.0% coverage within 1,000 steps for it on Matterport3D.
The direct competitor on the tuning question is S-BKI with a hand-tuned shared length-scale, run two ways. The first uses the published-style single value. The second is a stronger version: the single best fixed value found by grid search on a held-out set of environments from the training split, with the same total number of evaluations our method spends online. Beating the grid-searched fixed setting is what counts. S-BKI's paper reports IoU superiority over prior semantic mappers on KITTI and SemanticKITTI, but the file gives no numerical IoU value. We will reproduce its numbers ourselves rather than quote one.
The observation-fit tuning baselines are three. The first is kernel length-scale chosen by maximizing fit or likelihood on observed cells. The second is Bayesian rank pruning in the style of Bayesian Robust Tensor Ring Decomposition for Incomplete Multiway Data, which learns ranks by variational inference but only ever validates against observed entries on static benchmarks. The third is the weight-free min-max completion of Hybrid Singular Value Thresholding for Tensor Completion, which removes mode weights by minimizing the largest unfolding rank and was tested only on random synthetic tensors. All three get the same local tensor, the same compute budget per frame, and no access to ground-truth hidden cells. Beating them on extrapolation accuracy at distances beyond 1 m from the frontier is the central test of the core claim.
The strongest existing hallucinator is UPEN's ensemble of UNets trained to predict unobserved occupancy. On Matterport3D it reports 141.6 m² map accuracy and 67.9% coverage within 1,000 steps, and it raises Val-Hard point-goal success from 38.0% to 45.7% (Uncertainty-driven Planner for Exploration and Navigation). We run it with its released training regime on the same Matterport3D splits and plug its predictions into the same planner. We do not expect training-free completers to beat it in-distribution. Beating it means one of two things: matching it within a stated margin while using no training data and far less compute, or beating it on a shifted test set (train on Matterport3D, test on KTH floor plans) where its learned prior is out of distribution.
Two ablations isolate the key ingredient. The first replaces occlusion-shaped masks with uniformly random masks of the same total size, keeping everything else fixed. If random masks select equally good $\theta$, the core claim is false. The second shares one hyperparameter across all classes, which separates the gain from class-specific tuning from the gain from structured validation.
Why not just…?
Why not just grid-search the length-scale once per dataset and ship it? This is the most likely reason the project could turn out unnecessary. If environments within a deployment are homogeneous, a single well-tuned constant may be enough. The sensitivity-curve study is designed to settle it. We measure the gap between the best fixed setting and the per-environment oracle, across the dungeon and KTH floor plans used by CogniPlan and across Matterport3D houses. If that gap is small (under about two IoU points at the 1 to 3 m extrapolation band), we will report that hand-tuning suffices and the online method is not worth its complexity. The CogniPlan authors' own note that fixed-size grids limit transfer across geometric scales suggests the gap will be large under cross-dataset shift, but that is a hypothesis.
Why not just use marginal likelihood, or Bayesian rank pruning, which already tune automatically? These criteria reward explaining the cells already seen. That is the correct target only when missing cells are exchangeable with observed ones. BRTR and HSVT were validated only on randomly masked images, video, hyperspectral cubes, and synthetic tensors, so nothing in their evidence speaks to occlusion. Our baseline comparison tests this directly. If they track the oracle as well as structured validation does, we drop our selector in their favor.
Why not just train a deep hallucinator and forget about hyperparameters? Deep generators move the tuning problem rather than removing it. They still need a threshold for trusting a hallucinated cell, and they cost far more. Campbell et al.'s GAN takes about 5 seconds per frame on a high-end GPU and reaches inpainting mIoU of 13.10 on SemanticKITTI. PLMD's diffusion step takes 10 to 20% of inference time. The learned prior is also tied to its training distribution. UPEN is a real threat in-distribution, and we expect to lose there. The proposal's value rests on the latency comparison and the cross-dataset test. As a side experiment, the same occlusion-shaped selection can set UPEN's trust threshold, which would show that the principle is not specific to training-free completers.
Why not just simulate occlusion masks and skip the delayed-verification loop? This may well be the right answer, and we treat it as a co-equal arm rather than a strawman. Self-masking is immediate and cheap. Delayed verification is exact but lags by however long the robot takes to see a guessed region. We compare the two arms. If self-masking matches delayed verification, the simpler arm becomes the method.
Proposed Experiments
Map-level experiments run on three settings. The first is KTH floor plans and procedurally generated dungeons with simulated 2D LiDAR. The second is Matterport3D in Habitat with depth projected to an overhead grid. The third is SemanticKITTI for outdoor semantic classes, following S-BKI's setup.
The primary metric is extrapolation accuracy in distance bands beyond the observed boundary (0 to 1, 1 to 3, and over 3 m), reported as per-class IoU. We also report two asymmetric error rates. False-free is a wall predicted as open, which is the dangerous hallucination. False-blocked is open space predicted as a wall, which is the costly one. Calibration of predicted probabilities is reported as well. Each hyperparameter selector is also scored by the distance between its chosen $\theta$ and the per-environment oracle.
Navigation-level experiments use point-goal navigation on Matterport3D Val-Hard, where UPEN's 45.7% success is the reference. They also use exploration under a 1,000-step budget, measuring coverage and map accuracy against UPEN's 67.9% and 141.6 m², plus travel length and replanning count on KTH. Every method reports wall-clock time per frame on a laptop CPU.
The decisive checks are three. First, the random-mask ablation must select measurably worse $\theta$ than occlusion-shaped masks at the 1 to 3 m band. If it does not, the core claim fails. Second, per-class occlusion-tuned S-BKI must beat the grid-searched fixed S-BKI on the cross-dataset split, both in IoU and in false-free rate. Third, the low-rank completer with tuned rank must at least beat the optimistic planner in travel length at under 100 ms per frame. If it cannot, low-rank structure is not a usable prior for occupancy, and we report that as a negative result for the tensor arm while keeping the kernel arm.
References
- Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping
- Completion of High Order Tensor Data with Missing Entries via Tensor-train Decomposition
- Tensor Completion Using Bilayer Multimode Low-Rank Prior and Total Variation
- Accelerated and Inexact Soft-Impute for Large-Scale Matrix and Tensor Completion
- Tensor Completion for Radio Map Reconstruction using Low Rank and Smoothness
- DULRTC-RME: A Deep Unrolled Low-rank Tensor Completion Network for Radio Map Estimation
- Planning Paths Through Unknown Space by Imagining What Lies Therein
- CogniPlan: Uncertainty-Guided Path Planning with Conditional Generative Layout Prediction
- Uncertainty-driven Planner for Exploration and Navigation
- Bayesian Robust Tensor Ring Decomposition for Incomplete Multiway Data
- Hybrid Singular Value Thresholding for Tensor Completion