A robot that learns by trial and error in the real world cannot afford to crash in order to find out where crashing begins. Safe reinforcement learning methods therefore keep the robot away from danger, either with a constraint on expected cost or by handing control to a conservative fallback policy whenever the robot looks risky. A recent result exposes a cost of this caution. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that as a safe learner improves, it stops visiting unsafe states. The model it uses to judge "is this action recoverable?" then stops receiving examples of failure, its estimate of the boundary degrades, and safety costs spike late in training. That paper's fix is a second, adversarial policy that seeks out violations deliberately. This is acceptable in simulation and destructive on a physical robot.
We propose to take the failure examples from somewhere else. Mobile robots now have access to thousands of hours of passive, map-free navigation data: crowdsourced teleoperation logs, action-free first-person videos, and multi-robot datasets. Learning to Drive Anywhere with Model-Based Reannotation already turns such data into clean action labels using a short-horizon learned dynamics model. Our idea is to run that same kind of model counterfactually. For each passive frame we ask what would have happened under nearby alternative actions, and we keep the alternatives predicted to collide as negative examples. These "borrowed near-misses" form a fixed external pool of boundary data. A camera-only robot exploring without a map can then keep its sense of where danger begins sharp without ever producing a violation itself.
Core Claim
The safety paradox comes from where the boundary data originates, not from safe exploration as such. A feasibility estimator is a learned classifier that predicts whether a state-action pair can still be kept within the safety budget. If it is trained on a mixture of the robot's own safe experience and counterfactual violations mined from passive navigation data, its accuracy will not decay as the policy gets safer. A zero-violation learner equipped with it should match the late-training cost stability that the adversarial dual policy buys in simulation, and it should do so on hardware where the dual policy cannot run.
Concretely, let $F_\phi(o,a)\in[0,1]$ be the estimated probability that action $a$ taken from egocentric observation $o$ leads to an unrecoverable state. We train it on
$$\mathcal{L}(\phi)=\mathbb{E}_{(o,a)\sim\mathcal{D}_{\text{own}}}\big[\ell(F_\phi(o,a),y)\big]+\lambda\,\mathbb{E}_{(o,\tilde a)\sim\mathcal{D}_{\text{cf}}}\big[w(o,\tilde a)\,\ell(F_\phi(o,\tilde a),\tilde y)\big],$$
where $\mathcal{D}_{\text{own}}$ is the robot's own transitions with labels $y$, $\mathcal{D}_{\text{cf}}$ is the counterfactual pool of passive frames $o$ paired with perturbed actions $\tilde a$ and model-predicted outcomes $\tilde y$, $\ell$ is cross-entropy, $\lambda$ is a mixing weight, and $w$ down-weights counterfactuals where an ensemble of dynamics models disagrees. That weighting is the part that matters, because the reannotation literature itself flags dynamically infeasible labels from inaccurate forward models as its main weakness.
The discrete-action direction enters in one specific place: how the shield (the runtime safety filter) decides. Formal shields and temporal-logic verifiers stay tractable only over finite action sets, while smooth navigation needs continuous commands. We therefore let a diffusion navigation policy in the style of NoMaD propose a finite set of $K$ candidate continuous trajectories. The shield scores this finite set with $F_\phi$ and executes the best candidate that passes, falling back to a conservative prior policy only if none does. Verification happens over a discrete set, and execution remains continuous.
Positioning Against Prior Work
Two lines of work motivate the proposal by exposing the gap. The adversarial dual-policy paper (also here) diagnoses boundary-data starvation and cures it with deliberate violations. Our proposal keeps the diagnosis and replaces the cure with data that costs no hardware. Safe Exploration via Policy Priors, known as SOOPER, achieves zero training violations on a physical race car by falling back to a safe prior whenever predicted risk rises. Its zero-violation regime is exactly the one the paradox predicts will starve a boundary estimator over long deployments. We use SOOPER's fallback structure as the scaffold and ask whether its estimator drifts.
Two further works supply method. Model-based reannotation provides the short-horizon differentiable dynamics model and the passive-data pipeline; we use that model for counterfactual negatives rather than for optimal positive labels. NoMaD supplies a map-free, image-conditioned diffusion policy with topological memory, which we use as the source of candidate trajectories. Neither was trained with any safety constraint, and NoMaD still logs collisions in clutter.
Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation, called CROP, is the closest mapless-safety relative. It grows interval danger zones from collisions the robot actually experiences, over 2D LiDAR. Our proposal differs in two ways: its danger data is gathered before any collision, and it works from camera images, which the CROP authors list as unsolved. HASARD constrains the design. Its finding that privileged depth raises reward by 34% and segmentation by 21% indicates that spatial perception, not the optimizer, limits vision-based safety. This is a reason to expect that representations pretrained on large navigation data will help the estimator, and a reason to test that expectation directly rather than assume it.
Research Program
The work has three stages.
The first stage is counterfactual mining. We train an ensemble of short-horizon image-conditioned dynamics models on the GNM and SACSoN multi-robot logs used by ViNT and NoMaD, following the reannotation recipe. For each frame we sample action perturbations around the logged or reannotated action, roll the models forward a short horizon, and label outcomes by predicted clearance. Human interventions and actual contacts in crowdsourced logs serve as real negatives where they exist.
The second stage is the estimator and shield. $F_\phi$ shares an image encoder with the navigation policy. The shield filters $K$ diffusion samples, and the fallback prior is the pretrained policy run slowly with an inflated stop distance.
The third stage is online learning. The robot fine-tunes the policy within a SOOPER-style pessimistic-expansion loop on a new site with no prior map, and $\mathcal{D}_{\text{cf}}$ stays fixed throughout.
The central analysis tracks, over training, the fraction of boundary examples the estimator sees and the estimator's calibration on a held-out set of true near-misses.
Baselines
The naive reference is PPO with a Lagrangian cost constraint trained online from images, the standard constrained method. Safety Gymnasium reports that it cuts violations by 98% relative to PPO at a 45% reward drop on velocity tasks. It runs in simulation only, with the same encoder and step budget. Its role is to show that safety without pretrained structure is costly in both reward and violations.
The second baseline is the pretrained navigation policy deployed as-is, without fine-tuning. For it, NoMaD reports a collision rate of 0.2 per experiment versus 1.7 for its best baseline, and reannotation-trained LogoNav reports a 0.857 long-horizon goal success rate versus 0.333 for NoMaD. We run LogoNav-style reannotated training on the same passive pool as our method, so the only difference is the safety machinery. Beating it means fewer collisions at equal or better success on the new site.
The strongest competitor for the hardware question is SOOPER using the same pretrained prior and the same world-model ensemble, but with its feasibility estimate trained only on the robot's own data. On a physical race car SOOPER reports a policy twice as good as its prior with no collision during training. This baseline isolates our key ingredient exactly. It gets identical data, compute, and tuning trials for $\lambda$-equivalent hyperparameters. Beating it means matching reward with lower late-training cost or fewer fallback triggers, together with better estimator calibration after long deployment.
In simulation only, we add the adversarial dual-policy method as an oracle-like upper reference. It obtains real boundary data by crashing; its reported violation-sample ratio is about ten times that of its non-dual variant, which falls below 1%. Our method should approach its late-training cost stability without its violations.
Ablations remove the counterfactual pool, replace counterfactuals with randomly labeled perturbations (testing whether any extra negatives suffice), drop the ensemble-disagreement weighting, and replace the finite-candidate shield with a gradient-based projection of a single continuous action.
Why not just…?
Why not just use the pretrained navigation policy as SOOPER's safe prior and stop there? This is the most serious threat, and it is the main baseline. If the robot's own-data estimator never drifts within realistic deployment lengths, our addition is unnecessary. The decisive experiment runs both variants for long horizons in a MetaDrive environment built from imported real logs and measures whether late-training cost spikes or fallback-rate creep appear in the plain variant. The paradox paper predicts they will, but it showed this only for low-dimensional states. If no drift appears with image observations, we report that negative result as a finding about the paradox's reach.
Why not just penalize collisions in the reward? MetaDrive found that scalar penalties either let agents accept occasional catastrophes or produce agents that freeze. In any case, a penalty requires experiencing the collision on hardware, which is what we are trying to avoid.
Why not just learn the boundary in simulation with the dual policy and transfer it? MetaDrive reports that procedural diversity alone does not close the gap to real maps, and HASARD's results tie vision-based safety to perception quality, which is precisely what sim-to-real breaks. We test this directly. A dual-policy estimator trained in simulation and frozen is an extra arm on the real-robot site.
Why not just shield with a depth sensor and a stop distance? This is a strong practical competitor for collision avoidance, so it is our fallback prior and also a standalone arm. It cannot tell an untraversable patch from a drivable one in vegetation or on curbs. We expect it to win on raw collisions but lose on success and progress. Whether our method wins on the joint metric is an empirical question, and we treat it as one.
Proposed Experiments
The first decisive check is label fidelity. In MetaDrive with Waymo- and Argoverse-imported scenarios, the simulator provides ground truth for every counterfactual. We measure the precision and recall of mined negatives as a function of horizon and ensemble disagreement. This number sets $\lambda$ and the horizon before any hardware run. If precision is poor at useful horizons, the method fails here, cheaply.
The second check is the paradox curve. On HASARD's camera-only tasks and on MetaDrive (where it reports about 80% success on unseen maps with enough training scenarios), we plot, over training, the boundary-example fraction seen by the estimator, its calibration on held-out near-misses, and episode cost, for all baselines. The prediction is that the own-data variant's calibration decays while ours stays flat, and that our cost curve approaches the dual policy's without its violations.
The third check moves to hardware. A wheeled rover with a single camera fine-tunes at a campus site absent from the passive data, with no map, starting from the pretrained policy. Metrics are training-time violations (target zero), fallback-trigger rate over time, goal success, and progress per hour. The comparison is against the three real-world arms: the pretrained policy as-is, SOOPER with own-data estimation, and the depth stop rule.
The shield ablation reports per-step latency for $K$ candidates on embedded compute together with velocity smoothness, testing whether the finite-candidate design is affordable. A result in which counterfactual negatives improve calibration but not cost or success would mean the estimator was never the bottleneck, and we would say so.
References
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Learning to Drive Anywhere with Model-Based Reannotation
- NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration
- Safe Exploration via Policy Priors
- Online Safety Property Collection and Refinement for Safe Deep Reinforcement Learning in Mapless Navigation
- HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents
- ViNT: A Foundation Model for Visual Navigation
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning