Robots that navigate unseen buildings using only an RGB camera now do so by chaining pretrained vision models. One estimates depth from a single image, another segments the scene, a third decides which surfaces are floor. The result is a top-down map of free space that a planner drives through. These pipelines generalize to new layouts without training, but their perception errors are uncalibrated, and nothing bounds how often the robot drives into an obstacle the map missed. Methods that do certify safety in new environments need hundreds of calibration environments or perfect knowledge of the robot's state.
We propose a different route. The safety margin is calibrated around the perception pipeline itself, using conformal prediction (a distribution-free way to turn a few held-out examples into an error bound that holds with a chosen probability). The key choice is to treat each whole building as one calibration example rather than each camera frame. We measure how much the pipeline overestimated free space anywhere near the robot during one run, take a quantile of that number across roughly twenty buildings, and inflate every obstacle by that amount at test time. The claim is that, in a fresh building, the inflated map is conservative everywhere the robot can reach with probability at least $1-\alpha$. This holds from about $1/\alpha$ buildings, not hundreds. It holds under whatever sensor noise was present during calibration. And it needs no collisions to learn, because the calibration signal is map error measured against true geometry, not crashes.
Core Claim
Let $\hat d_t(x)$ be the free-space distance the foundation-model pipeline predicts at map location $x$ at time $t$, and $d(x)$ the true distance to the nearest obstacle. For a run in building $e$, define the building-level score
$$s_e = \max_{t} \; \max_{x \in R_t} \big(\hat d_t(x) - d(x)\big),$$
where $R_t$ is the set of locations the robot could reach within its braking horizon from its pose at time $t$. The score is the worst overestimate of clearance anywhere the robot could have gone. Given $n$ calibration buildings, let $q$ be the $\lceil (n+1)(1-\alpha)\rceil$-th smallest of $s_1,\dots,s_n$. If a new building is exchangeable with the calibration set, then with probability at least $1-\alpha$ the true clearance in it satisfies $d(x) \ge \hat d_t(x) - q$ on every reachable location at every step. This is the standard split-conformal guarantee, applied with buildings as the exchangeable units. A shield that vetoes any action whose swept footprint leaves less than $q$ plus braking distance of predicted clearance, falling back to stopping and rotating in place, then inherits a building-level collision guarantee.
Three things are new relative to the closest work. The guarantee is closed-loop and spatial, not a passive warning. It is calibrated per building, which absorbs the strong frame-to-frame correlation that would break per-frame exchangeability. And its cost in calibration buildings scales as $1/\alpha$, following the $\Theta(1/\epsilon)$ conformal rate.
Position Relative to Prior Work
Two papers motivate the gap. Sim-to-Lab-to-Real trains a safety critic from RGB via reachability analysis and a backup policy that intervenes when that critic predicts a violation. It certifies generalization to new 3D-FRONT houses with PAC-Bayes bounds, but non-vacuous bounds require evaluating hundreds of lab environments. We replace the PAC-Bayes certificate with a conformal one over far fewer buildings. We also certify the perception module rather than a learned policy distribution, so the certificate survives swapping the planner.
Sample-Efficient Safety Assurances using Conformal Prediction shows that conformal calibration of collision warnings needs only $\Theta(1/\epsilon)$ unsafe examples, versus $\Theta(1/\epsilon^2)$ for PAC learning. It also shows that the certificate stays intact when predictors degrade under noise, at the cost of more false alarms. That work is an open-loop warning system on trajectory forecasting and grasping. We supply the method from it, and our contribution is making it a closed-loop navigation shield with buildings as the calibration unit.
Two other lines constrain the design. ReCORS builds conformal uncertainty tubes around learned dynamics but assumes the true state is known. SOOPER falls back to a safe prior policy using ensemble uncertainty, which unmodeled sensor noise can miscalibrate, and it assumes episodic resets. Both tell us that the uncertainty must be placed on perception, where the noise enters, and must not rely on ensemble variance. We therefore put the conformal bound on the observation-to-map step and keep the robot's motion model simple and conservative.
Provably Safe Sim-to-Real Transfer achieves zero unsafe iterations by concentrating real data on simulator-mismatch regions, but only in a tabular 5×5 gridworld. Its insight that real data should be spent only where simulation is wrong reappears in our recalibration step below.
The zero-shot navigation systems supply the pipeline we wrap. TANGO builds local top-down cost maps from SAM, CLIP and Depth-Anything. Its fallback to reactive visual servoing is heuristic, and it still collides when CLIP labels furniture as floor. ESC and PixNav likewise have no physical safety constraint. We do not change how these systems choose goals; we bound how wrong their free-space estimates can be.
Finally, Breaking Safety Paradox with Feasible Dual Policy Iteration observes that safe learners stop seeing violations and so cannot estimate where the safety boundary lies. Our score sidesteps this, because it measures clearance error and needs no contact.
Research Program
The first component is the wrapped pipeline. We start from TANGO's RGB-to-top-down map. Ground-truth clearance $d(x)$ comes from the simulator mesh in calibration buildings, and from a temporarily mounted depth sensor or prebuilt scan in physical calibration buildings. The scan is used only for calibration, never at test time.
The second component is the reachable set $R_t$. It is the robot footprint swept over its braking horizon from the current pose and velocity, computed from a conservative unicycle model. Restricting the score to $R_t$ rather than the whole image keeps $q$ from being inflated by distant depth errors that never matter.
The third component is conditional margins. A single $q$ will be loose for open hallways and tight near glass and thin chair legs. We will test Mondrian conformal prediction, meaning separate quantiles per category, here per predicted-distance bin and per CLIP semantic class. Each category keeps its own coverage, and the cost is more calibration buildings per category.
The fourth component handles a real subtlety: the score depends on which viewpoints the calibration runs visited, and the shield changes those viewpoints. We address this by collecting calibration runs with a deliberately diverse exploration policy and inflating $R_t$ to include all headings. We then test empirically whether coverage holds when the deployed policy differs.
The fifth component is sensor noise. Exchangeability breaks when calibration uses clean simulated images and deployment sees motion blur, gain changes, and lens flare. Our response is that, because calibration needs only around twenty buildings, it is affordable to calibrate under the target noise: real buildings, or simulation with the robot's measured corruption profile. We will quantify exactly how much coverage is lost when this step is skipped.
The sixth component is safe exploration and sample efficiency inside a new building. Once the shield is in place, a learner may adapt the local planner online. We use FOCOPS, which handles constrained policy updates with first-order steps, and the reward is progress toward topological subgoals. The shield turns that exploration from trial-and-crash into trial-and-veto. Because veto events carry no collision, the learner's constraint-cost signal is the shield-intervention rate. The question is whether a few hundred shielded episodes recover the success lost to conservatism.
Baselines
The naive reference is unmodified TANGO with its auto-switch fallback, run on the same HM3D trajectories with the same foundation models. TANGO reports a 43.14% success rate on hard trajectories without ground-truth metric information, rising to 90.20% with ground-truth metric depth (TANGO). The file reports no collision rate for it, so we will measure that ourselves. Beating this reference means a building-level collision rate at or below $\alpha$ with a stated success cost. The ground-truth-depth configuration serves as the upper bound on what any margin can recover.
The second baseline is a fixed obstacle inflation, a classical safety buffer, with its radius tuned on the same twenty calibration buildings to hit the same empirical collision rate. This is the strongest simple competitor and the honest test of whether conformal calibration adds anything beyond "pick a margin on held-out data."
The third baseline is an ensemble-variance shield in the style of SOOPER, using five depth heads with margin equal to $k$ standard deviations. Here $k$ is tuned on the same buildings, and the compute budget matches ours.
The fourth baseline is the strongest certified method, Sim-to-Lab-to-Real, run on 3D-FRONT where its numbers exist. It reports certified safety bounds improving from 0.304 to 0.708 and success bounds from 0.297 to 0.701 in its Advanced-Realistic setting, using hundreds of lab environments. We will plot our certified safety level against the number of calibration environments and report the environment count at which we match a 0.708 safety guarantee. Beating it means matching that guarantee with an order of magnitude fewer environments at comparable success. The comparison is caveated by the fact that its bound is a PAC-Bayes lower bound on a policy distribution while ours is marginal conformal coverage.
Two ablations isolate the key ingredient. The first calibrates per frame rather than per building. We expect it to under-cover because correlated frames violate exchangeability, which would show that the building-level unit is doing the work. The second drops the reachable-set restriction, which we expect to inflate $q$ and collapse success.
Why not just…?
Why not just inflate obstacles by a fixed, hand-tuned margin? This is the real threat. A margin tuned on held-out buildings to a target collision rate is already a crude quantile. If it matches our safety and success, the contribution shrinks to a principled tuning procedure plus a guarantee. The guarantee still matters for certification, but it is a smaller claim. Two experiments settle the question. Coverage curves under held-out noise profiles test whether a margin tuned under one condition fails silently. Conditional conformal margins test whether per-class and per-distance quantiles buy success that a single radius cannot, near furniture that CLIP misreads. The ROS-stack comparison in Deep Local Trajectory Replanning and Control (66% success on an unseen map versus 84% for a learned replanner) suggests that rigid buffers cost success in clutter, but that is not direct evidence about calibrated buffers.
Why not just train an end-to-end safe policy with domain randomization? Randomization can make a policy empirically robust, but it gives no statement about a new building. Sim-to-Lab-to-Real needed reachability shielding on top of diverse simulated houses, reporting 4% to 77% fewer training violations than SQRL and Recovery RL. We include a domain-randomized constrained policy (PPO-Lagrangian trained across 3D-FRONT with the same corruptions) as an additional baseline. If its realized collision rate on held-out buildings already sits below $\alpha$, our method's value reduces to certifying it.
Why not just use a depth sensor or LiDAR instead of monocular depth? On many robots one should, and our method applies equally to LiDAR-derived maps. The reflection and absorption artifacts documented for real LiDAR still leave free-space errors to calibrate. We keep RGB as the main setting because that is where zero-shot pipelines like TANGO operate and where errors are largest. We will run the LiDAR variant to show that $q$ shrinks rather than vanishes.
Why not just use ensemble uncertainty? Ensembles are cheap. But their variance has no coverage guarantee and can be miscalibrated by unmodeled noise, the failure mode flagged for SOOPER. The matched-compute ensemble baseline above tests this directly, under both clean and corrupted inputs.
Proposed Experiments
The main benchmark is HM3D with TANGO's easy and hard trajectory splits. Calibration buildings are disjoint from test buildings. We sweep $n \in \{5, 10, 20, 50\}$ and $\alpha \in \{0.05, 0.1, 0.2\}$.
Metrics are building-level collision rate, empirical coverage of the clearance bound, success rate and SPL (success weighted by path length), shield-intervention rate, and freeze rate (episodes ending stuck). We add RGB corruptions at test time (blur, gain shifts, flare, noise) in three regimes: calibrated under the same corruptions, under clean images, and under a different corruption family. 3D-FRONT hosts the Sim-to-Lab-to-Real comparison. A small physical study in five to ten real buildings, with five calibration buildings scanned in advance, tests the sim-to-real case.
Three decisive checks determine whether the idea stands. First, building-level coverage must hold at roughly $1-\alpha$ on held-out HM3D buildings with $n \approx 20$, while per-frame calibration measurably under-covers. Second, calibrating under clean simulated images must break coverage under real-style corruption, and recalibrating on a handful of noisy buildings must restore it. This is the evidence that cheap calibration is what makes sensor-noise robustness tractable. Third, success must stay within a modest margin of unshielded TANGO's 43.14% on hard trajectories while collisions drop to the certified level. Shielded FOCOPS adaptation should close part of the remaining gap within a few hundred episodes and produce zero collisions while learning.
If the fixed-margin baseline matches us on all three checks, we will report that outcome plainly. The paper then becomes a careful demonstration that a small, calibrated buffer suffices for certified zero-shot navigation.
References
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Sample-Efficient Safety Assurances Using Conformal Prediction
- Conformal Reachability for Safe Control in Unknown Environments
- Safe Exploration via Policy Priors
- Provably Safe Sim-to-Real Transfer
- TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goals
- ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation
- Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skill
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- First Order Constrained Optimization in Policy Space
- Deep Local Trajectory Replanning and Control for Robot Navigation