Gating Sim-Trained Imitation Policies on the Map's Own Doubt

A robot policy learned by imitating demonstrations in simulation will drift once it runs on real hardware. Small action errors move it into states the demonstrations never covered, and real sensors and terrain differ from the simulated ones. Interactive imitation methods respond by handing control to a human when the policy looks unsure. The trouble is that the policy's own uncertainty is a poor alarm: a policy that never saw a hazard class in simulation can be confident and still be wrong. This proposal moves the alarm out of the policy and into the robot's map. The map is built with probabilistic semantic mapping, meaning each 3D cell keeps a probability distribution over labels such as "ground," "mud," or "obstacle" instead of a single hard label. Before the robot executes its next short sequence of actions, we ask how likely it is, under the map's belief, that the space the robot will sweep through contains a hazard. If that probability is too high, the robot falls back to a safe action and either looks again or asks a human for a correction. The corrections are added to the training set, as in dataset aggregation.

This matters because it brings three problems under one mechanism. First, it gives a safety check that is defined over a sensor-fused, persistent representation, which should transfer from simulation to reality better than a critic reading raw pixels. Second, it targets interventions at the places where the policy's training data was thin and the world was uncertain, which is the region where compounding imitation errors start. Third, it supplies expert labels near safety boundaries, which is exactly the region that over-cautious safe learners stop visiting.

Core Claim

We claim that a chance constraint computed from per-cell semantic posteriors over the volume the next action chunk will sweep is a better trigger than policy-side uncertainty, for both stopping and querying, when a simulation-trained imitation policy is deployed under perceptual and semantic shift. "Better" here means fewer safety violations at a matched human-query budget. We further claim that splitting the trigger into two cases improves on a single uncertainty number. The first case is "the map is unsure because it has little evidence," and the robot should look again. The second is "the map is fairly sure this is a hazard and the policy is heading into it anyway," and the robot should ask the expert. This split should cut human queries without raising violations.

Concretely, we use a mapper that keeps Dirichlet concentration parameters $\alpha_{v,k}$ for each voxel $v$ and class $k$. A Dirichlet is a distribution over class-probability vectors, and its total $\alpha_{v,0}=\sum_k \alpha_{v,k}$ acts as an evidence count. Summing over the hazard classes $H$ gives a hazard mass $\theta_v \sim \mathrm{Beta}(\alpha_{v,H},\,\alpha_{v,0}-\alpha_{v,H})$. For a candidate action chunk $a$ with swept volume $S(a)$, the gate score is

$$U_\delta(a) = 1 - \prod_{v \in S(a)} \bigl(1 - Q_{1-\delta}(\theta_v)\bigr),$$

where $Q_{1-\delta}$ is the upper $(1-\delta)$ quantile of the Beta. Unobserved voxels keep the prior, so they score as risky by construction. The robot executes $a$ only if $U_\delta(a)\le\eta$, with the threshold $\eta$ set to respect a human-intervention budget. When the gate fires, the cause is read from the evidence: if the high-scoring voxels have low $\alpha_{v,0}$, the robot pauses to observe; if they have high $\alpha_{v,0}$ and high hazard mass, it queries the expert. The product treats voxels as independent. That is optimistic, but it is cheap, and its calibration is something we measure directly rather than assume.

Position Relative to Prior Work

Three papers supply the mechanism. ConvBKI recasts Bayesian kernel inference as a differentiable convolution that keeps per-voxel Dirichlet parameters at real-time rates. It reports 46.6% mIoU when transferring from simulated CarlaSC to real data, while implicit memory networks drop by nearly 90%, and it shows variance spikes on unmodeled objects such as a tractor. It never connects that variance to action selection, and that connection is our contribution. ThriftyDAgger gates human help using ensemble action variance for novelty and a learned failure-risk estimate, with thresholds tuned to a budget. We keep its budget-and-hysteresis machinery but replace the source of the signal. DAgger supplies the aggregation loop that turns quadratic error growth into linear growth.

Three papers motivate the proposal. Blind Spot Detection for Safe Sim-to-Real Transfer shows that agents missing safety-relevant features are confidently wrong. Its learned query classifier works only in low-dimensional toy games built on hand-designed features, and it is more accurate when trained on suboptimal, boundary-visiting data (AUC 0.96) than on optimal data (AUC 0.60). DropoutDAgger gates expert takeover on dropout-based action uncertainty with hand-tuned thresholds, and it has no spatial memory. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safer learners stop collecting near-boundary data. Our expert-query branch is a direct, if partial, answer to that: it produces safe labels at the boundary.

Two papers constrain the design. Sim-to-Lab-to-Real shields reinforcement-learning navigation with a reachability critic computed from RGB images. It reaches 76.7% success and 86.7% safety on a real quadruped, but it needs an intermediate lab stage with many environments, and its critic has no persistent semantic memory. Provably Safe Sim-to-Real Transfer gets zero unsafe iterations, but only in tabular problems and under a separable-mismatch assumption that continuous perception violates. We give up formal guarantees in exchange for a continuous, calibratable gate. We also inherit two limits we do not resolve. ConvBKI's static-world assumption confines us to static scenes. And if the upstream segmenter is confidently wrong, as it often is on RELLIS-3D, where image mIoU falls to about 51.55%, then the Dirichlet will be confidently wrong too. Measuring how often this happens is one of the main outputs of the work.

Research Program

We build three things. The first is a navigation imitation policy that takes an egocentric crop of the ConvBKI semantic map and outputs short action chunks. It is trained by behavior cloning on simulated expert demonstrations. The second is the swept-volume gate above, implemented on the GPU over the same voxel grid. The third is the two-branch fallback: observe-in-place for low-evidence triggers, and expert query plus safe stop for confident-hazard triggers. The query branch feeds a DAgger loop. All labels collected during deployment are aggregated and the policy is retrained between rounds.

The analysis has three parts. First, we check calibration of $U_\delta$ against ground-truth hazard intersection, both on real logged data and in simulation. Second, we trace a violation-versus-query-budget curve for each gating method. Third, we break violations down into those caused by low map evidence, those caused by confident misclassification, and those caused by policy error on a correctly mapped hazard. That breakdown tells us when map-side gating cannot help.

Baselines

The naive reference is plain behavior cloning on simulated demonstrations with domain randomization of rendering and sensor noise. It is deployed with no gate and no fallback, using the same map input, data, and compute. Any method that does not reduce violations relative to this reference has failed.

The strongest existing competitor is ThriftyDAgger, which combines ensemble novelty with learned risk and sets its thresholds by budget. We run it with the same map-input policy, the same simulated demonstrations, the same ensemble size and compute, and the same intervention budget, and we tune its two thresholds over the same grid we use for $\eta$. Its reported figure is 73% autonomous success on simulated peg insertion, against 24% for SafeDAgger and 48% for LazyDAgger, as given in the ThriftyDAgger paper. Its user study also reports human and robot performance gains of 58% and 80% over the next-best algorithm. All of these are manipulation results; we found no reported result on Lacuna for ThriftyDAgger on semantic-map navigation, so we report our own reimplementation. To count as beating it, our gate must achieve significantly fewer violations at every matched query budget along the curve, with no loss in task success.

The closest navigation system is the reachability shield of Sim-to-Lab-to-Real, which reports 76.7% success and 86.7% safety on real hardware and certified success and safety bounds of 0.701 and 0.708 in 3D-FRONT Advanced-Realistic. We run its image-based safety critic as a drop-in shield over the same behavior-cloning policy, without its lab stage, because our setting has none. We note that this comparison is not identical to its published setup.

A deterministic baseline converts the map to argmax labels and runs a geometric collision check on the swept volume with an inflated margin, tuned over a grid of margins.

Three ablations isolate the key ingredients. The first replaces the swept-volume Beta-quantile score with mean voxel entropy in a radius around the robot. This tests whether making the score depend on the planned action matters. The second uses the posterior mean instead of the upper quantile. This tests whether epistemic evidence counts matter. The third collapses the two branches into a single query trigger. This tests whether the split reduces human burden.

Why not just…?

Why not just feed the map to ThriftyDAgger and let its ensemble handle it? This is the real threat. An ensemble trained on map inputs may disagree exactly where the map is unfamiliar. Our argument against it is that ensemble members trained on the same simulated maps will agree on hazard classes or textures absent from simulation, which is the confident-and-wrong pattern documented by Blind Spot Detection. The map's evidence counts, by contrast, register "never observed well" whatever the training data contained. This argument is untested. The matched-budget comparison, broken down by violation cause, settles it. If ThriftyDAgger matches us on the low-evidence and novel-class splits, our claim is wrong and we will report that.

Why not just inflate obstacles in a deterministic map? Inflation with hard labels cannot tell a well-observed safe patch from an unobserved one. It cannot represent a misclassified hazard either: Stiller et al. show that argmax labeling passes errors silently to planners. Inflation can still be competitive in clutter-free scenes. For that reason we report the full trade-off between progress and violations as the margin varies, rather than a single tuned point, and we judge success by whether our curve dominates it.

Why not just use domain randomization and skip the gate? Randomization cannot vary a feature the simulator does not model. Neither Sim-to-Lab-to-Real nor the blind-spot work finds that randomization alone gives safety. The naive baseline measures this directly on held-out hazard classes.

Proposed Experiments and Decisive Checks

The first experiment checks offline calibration on real logs, with no closed loop. On SemanticKITTI and RELLIS-3D sequences, we build ConvBKI maps from a CarlaSC-trained network, sample candidate action chunks at each recorded pose, and test whether $U_\delta$ predicts intersection with ground-truth hazard voxels, measured by expected calibration error and AUC. We compare against ensemble variance from the policy fed the same maps. If the map-side score is not better calibrated on real logs, the closed-loop work is not worth doing, so this is the first decisive check.

The second experiment is closed-loop navigation in 3D-FRONT layouts, using the Advanced-Env split from Sim-to-Lab-to-Real. Training uses one rendering and class set. Testing injects sensor noise and lighting shift and introduces held-out hazard categories. We plot violations against queries per kilometer, and report success and number of DAgger rounds to reach a target success level.

The third experiment is a small real-robot deployment in static indoor clutter, reporting success, violations, and queries. It is framed against the 76.7% success and 86.7% safety reported for Sim-to-Lab-to-Real, with the caveat that the robots and environments differ.

The decisive result has two parts. At matched query budget, the swept-volume posterior gate must beat ThriftyDAgger with map input on violations within the novel-class and low-evidence splits. And the two-branch version must lower queries relative to the single-branch ablation without raising violations. If confident misclassification dominates the remaining violations, that finding sets the limit of map-side gating, and it points the next step toward calibrating the segmenter itself.