Learning Where Real Safety Boundaries Lie Without Crossing Them

A robot trained in simulation brings two things to the real world: a policy, and an implicit belief about which states are safe. When the simulator is wrong, the safety belief is wrong exactly where it matters most, at the edge between safe and unsafe. Fixing that belief normally requires failures. This proposal treats the problem as a sparse-signal problem. Safety violations are rare, binary events, much like a sparse task reward, and the cure is the same one that works for sparse rewards: replace the rare binary event with a dense, graded signal. A real robot can measure how close it came to a constraint (distance to an obstacle, torque headroom, how far a joint is from its limit) on every step without ever violating it. We propose to learn a feasibility function in simulation, then correct it on the real system using only these near-miss margins. Real-world probing is kept a calibrated distance inside the current estimate of the boundary, and that distance shrinks as evidence accumulates.

The result would settle a split in the literature. One camp argues that boundaries can only be mapped by visiting them, through violation-seeking policies or deliberately suboptimal demonstrators. The other holds real violations to exactly zero. Our claim is that the tension goes away once the safety signal is graded rather than binary. An agent can then learn where the real boundary sits by approaching it, without crossing it.

Core Claim

Define a margin $m(s)$, a continuous measurement that is positive in safe states and crosses zero at a violation. From it define a feasibility critic $V(s)$, the predicted minimum future margin under the current policy. In simulation, $V$ is trained with boundary-seeking data, so it is accurate near its own zero level set. On the real system, we never observe violations, but we do observe margins. The residual between the simulated critic's prediction and the realized minimum margin along short real trajectories is therefore a dense regression target.

We use conformal prediction, a distribution-free method that turns held-out residuals into an error bound with a chosen coverage level, to compute a quantile $q_\alpha$ of those residuals. The probing constraint on the real robot is

$$\hat V(s) - q_\alpha(s) \ge \delta,$$

where $\hat V$ is the sim-trained critic after real-data correction, $q_\alpha(s)$ is the conformal residual quantile at miscoverage level $\alpha$ (localized by state features), and $\delta \ge 0$ is a safety buffer. The critic is corrected by fitting a small residual model to real near-miss data, and as it improves, $q_\alpha$ shrinks. The prediction is that real-world violation frequency stays at or below $\alpha$ during probing, because that is the coverage guarantee conformal quantiles provide under exchangeability. We also predict that the corrected boundary is tight enough to recover task return that pessimistic zero-shot methods give up.

Sparse task reward matters here for a specific reason. When the goal reward is rare, a safe policy has no incentive to approach boundaries. It therefore collects grounding data far from where the simulator's safety model is wrong. Under the same pressure, Lagrangian methods drift toward the "safe but useless" frozen policy documented by Quanyi Li et al.. Directed, margin-limited boundary probing is thus most needed in the sparse-reward regime, and that is where we evaluate it.

Relation to Prior Work

Two papers motivate the gap by showing what fails. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that feasibility critics degrade as policies become safe, because violation data falls below 1% of the replay buffer. It fixes this with a KL-constrained dual policy that deliberately seeks violations. We reuse this technique, but only inside simulation. Running a violation-seeking policy on hardware is exactly what we must avoid, so the real-world stage replaces violations with margins. Blind Spot Detection for Safe Sim-to-Real Transfer makes the same point from the data side: blind-spot classifiers trained on suboptimal oracle data reach an AUC of 0.96, against 0.60 from optimal data, because optimal experts avoid boundaries. That method depends on a human oracle during deployment and was tested on low-dimensional games. Ours is autonomous and continuous.

Two papers constrain the design and are the closest prior work. Provably Safe Sim-to-Real Transfer finds the regions where the simulator is wrong with statistical confidence bounds. It builds a hybrid model and collects real data with a mixture of an exploratory policy and a safe baseline, reaching 0% unsafe iterations in a gridworld. However, it is tabular, and it assumes mismatch is either zero or above a detectable threshold. Our conformal residuals handle continuous noise without that separability assumption. We also correct a boundary rather than a full transition model, which keeps the problem tractable in continuous spaces.

Sim-to-Lab-to-Real already uses a dense margin through Hamilton-Jacobi reachability, a method that computes the set of states from which a constraint can still be avoided. Its correction comes from PAC-Bayes bounds over hundreds of lab environments, and it assumes the lab and real distributions match. We use the same kind of margin, but calibrate it online on the single real system being deployed.

SPiDR supplies our strongest baseline. It inflates the safety cost by the disagreement among an ensemble of randomized simulators, which gives zero-shot safety. It assumes real physics lie inside the randomization envelope, and it uses no real data to tighten the boundary afterward.

Research Program

The first stage happens in simulation. We train a task policy and a feasibility critic over the margin $m(s)$ with a constrained off-policy learner, adding a FDPI-style dual policy so the critic sees boundary data. The domain randomization is the same as SPiDR's, so the two starting points match.

The second stage probes the real system safely. A probe policy is rewarded for reducing $\hat V$ and for reaching states where the residual model is uncertain, subject to the conformal constraint above. A fixed backup controller from simulation takes over whenever the constraint would be breached, as in shielding approaches. Real trajectories are short, and their realized minimum margins update both the residual model and the conformal calibration set.

The third stage adapts. The task policy is fine-tuned against the corrected critic in simulation, with no real task rollouts beyond evaluation.

The validation methodology follows Reinforced Grounded Action Transformation: a modified simulator stands in for "reality." Its physics perturbations are chosen deliberately outside the randomization envelope, covering friction, actuator delay, mass, and added latency with quantization. In this surrogate the true boundary can be computed exactly, so errors in the boundary estimate become measurable. A small hardware study then checks that the surrogate results hold on a real robot.

Baselines

The naive reference point is domain randomization with a Lagrangian constrained learner and no real-world correction. This is the setting in which SPiDR reports 26.1x higher violations than its method on a Unitree Go1 and 6.2x higher on a 1/10-scale RaceCar. A second naive reference is the unconstrained sim-to-real fine-tuning baseline from Provably Safe Sim-to-Real Transfer, which violated constraints in 68.5% of learning iterations in its gridworld.

The strongest competitor is SPiDR itself, which reports satisfying constraints 100% of the time in its hardware tests. It is the strongest because it already achieves zero-shot safety cheaply. Beating it on violations alone is therefore impossible within its envelope. The contest is on two other axes. The first is task success at equal violation rates. We found no reported task-return number on Lacuna for SPiDR's hardware tests, so we will measure it ourselves. The second is safety when the surrogate physics fall outside the envelope. SPiDR will run with the identical ensemble and the same simulator step budget. Its penalty weight will be tuned with the same number of search trials we give our buffer $\delta$ and level $\alpha$.

Beating SPiDR means one of two outcomes. Either sparse-goal success is higher at violation rates no worse than SPiDR's inside the envelope, or violations are strictly fewer outside it, in both cases with at most a fixed real-sample budget.

We also compare against a grounding baseline: RGAT trained on real data collected under our backup shield with the same real-sample budget. This tests whether correcting the dynamics beats correcting the boundary.

Three ablations isolate the key ingredient. The first replaces the graded margin with a binary violation cost. Here calibration should fail because no residuals are ever observed, and the method should collapse to a fixed buffer. The second probes with the task policy instead of the boundary-directed probe policy, which tests whether directed data matters under sparse reward. The third removes the FDPI dual policy in simulation, which tests how much the starting critic's quality matters.

Why not just…?

Why not just use SPiDR's pessimism and skip real data? Inside the randomization envelope, this is a real threat. If the surrogate perturbations stay within the ensemble, SPiDR may match us on safety with zero real samples, and we would report that. Our bet rests on two points. First, pessimism set by ensemble disagreement cannot shrink where the real system turns out to be benign, so it gives up sparse-goal success. Second, it offers no protection when the real system lies outside the ensemble. The out-of-envelope surrogate and the matched-violation return comparison decide both points directly.

Why not just tighten the constraint by a fixed margin? A fixed buffer is our method with the residual quantile frozen at a constant. If a single tuned constant does as well, the conformal machinery is unnecessary. The comparison will sweep fixed buffers and plot success against violations. We expect the adaptive buffer to dominate the whole curve, because real mismatch is uneven across states: a buffer large enough for the worst region wastes the rest. Evidence that uniform conservatism is costly includes the 45% return drop that Safety Gymnasium reports for PPO-Lagrangian when it cuts violations by 98%.

Why not just ground the simulator with shielded real data, as in RGAT, and retrain? This is plausible, and it is the alternative we take most seriously. RGAT as published collects real transitions without any safety mechanism. With our shield added, its data would come from a sparse-reward policy that rarely approaches boundaries, so its dynamics correction may be accurate in the wrong places. The "RGAT on our probe data" variant separates the question of where the data comes from from the question of whether to model dynamics or the boundary. If that variant matches us, the contribution reduces to the probing strategy, and we would say so.

Why not just replay the few early violations with prioritized experience replay? The FDPI paper already shows that passive prioritized replay re-samples stale, out-of-distribution transitions and gives noisy feasibility estimates. On real hardware there would be almost no violations to replay in the first place.

Proposed Experiments

The main testbed is Safety-Gymnasium PointGoal and CarGoal, following SPiDR's PointGoal2 port. We use sparse-goal variants in which reward is given only on reaching the goal, and the margin is the signed distance to hazards. For each task we build three surrogate-real simulators, one inside the envelope and two outside it. The measurements are violations during real probing (count and per-episode rate), sparse-goal success and cost rate after adaptation, boundary error against the exact surrogate boundary (the area wrongly classified as safe or unsafe), and the number of real samples used. The hardware study uses a 1/10-scale RaceCar setup of the kind SPiDR used, with a real budget of a few hundred short probe trajectories.

Three checks are decisive. The first is coverage: the empirical violation rate during probing should stay at or below the nominal $\alpha$ in every surrogate. If it does not, the calibration claim is false. The second is dominance: at matched violation rates, our success should exceed SPiDR's and the best fixed buffer's inside the envelope, and our violations should be lower than SPiDR's outside it. The third is the margin ablation: replacing the graded margin with a binary cost should remove most of the gain. If it does not, dense margins are not the reason the method works, and the core claim fails.