A robot trained in a simulator usually performs worse, and less safely, once it meets real physics. Exploring in the real world can close that gap, but every exploratory mistake on hardware costs something. Current methods choose one of two fixed attitudes. They are either pessimistic everywhere, keeping the robot well inside the safe region so a wrong simulator cannot hurt it, or they assume the simulator is right and push the policy onto the exact edge of what is allowed, which is where the best constrained policies sit. This note proposes a third attitude. The robot keeps a safety margin that depends on the situation. The margin is measured, not guessed, from how wrong the simulator's safety predictions have actually been on nearby real experience. The robot starts cautious and is permitted to move closer to the constraint only in the parts of the state space where real data shows the simulator's safety estimates can be trusted.
Concretely, a safety critic is trained in simulation. A safety critic is a learned function that predicts how much constraint cost a policy will accumulate from a given state and action. That critic is then wrapped with a statistical correction learned from real rollouts using conformal prediction, a distribution-free method for turning observed prediction errors into error bars with a target coverage rate. Real-world exploration may take an action only if the critic's prediction plus the local error bar stays under the safety budget. Wherever there is too little real data to compute an error bar, control passes to a pessimistic policy or a known-safe backup policy. The new element is the measured, local margin. Prior work bounds simulator error either by how much simulators disagree with each other or by an exact certification that only exists for small discrete problems.
Core Claim
Real-world violations during adaptation are governed by how wrong the simulator is near the constraint boundary, not across the whole state space. Most of that error can be measured from safe real rollouts that come close to the boundary without crossing it. If the margin is set as a local conformal upper quantile of the critic's real-world error, violations during adaptation should stay near the chosen miscoverage level $\alpha$ even when the real dynamics lie outside the randomization range used in training. Final reward should approach that of a boundary-tracking policy wherever the simulator turns out to be accurate.
The deployment rule is
$$\hat Q_c(s,a) + m_t(s,a) \le d,$$
where $\hat Q_c(s,a)$ is the simulation-trained critic's prediction of $H$-step discounted constraint cost after taking action $a$ in state $s$, $d$ is the cost budget, and $m_t(s,a)$ is the margin at adaptation round $t$. The margin is the kernel-weighted $(1-\alpha)$ quantile of one-sided errors $C^{\text{real}}_H - \hat Q_c$ observed on earlier real segments near $(s,a)$. If the effective number of nearby real samples falls below a threshold $n_{\min}$, the pair is treated as uncertified and the backup policy acts. Using an $H$-step horizon keeps the calibration target directly observable on short real segments. Without it, the calibration target would be an infinite-horizon return that can never be fully observed.
Relation to Prior Work
The closest work splits into three roles: work that motivates this idea, work that constrains its design, and work that supplies parts of the method.
The motivation comes from Provably Safe Sim-to-Real Transfer. That paper shows, for discrete-state constrained problems, that the number of real samples needed scales with the size of the "mismatch region" where the simulator is wrong, not with the full state space. It certifies accurate state-action pairs with confidence bounds and explores using a mixture with a safe baseline policy. Its method is tabular and solved by linear programming. It also assumes the simulator is either exactly right or wrong by at least a detectable gap. We keep the idea of spending real samples only where the simulator is untrusted. We replace exact certification with certification of an error bound in continuous spaces, which drops the "exact-or-detectably-wrong" assumption. The price is a weaker, empirical coverage guarantee.
The constraints come from SPiDR. SPiDR adds the variance of an ensemble of randomized simulator models to the cost and achieves zero-shot safety on a Unitree Go1 and a 1/10-scale race car. It does no real-world learning, uses a fixed penalty weight, and its pessimism depends only on disagreement among simulators. When reality lies outside the randomization range, every simulator can agree and still be wrong, so the penalty disappears exactly where it is needed. Our margin is built from real errors, so it does not share that blind spot. SPiDR also serves as our initial uncertified behavior.
Several papers supply method components. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safety critics lose calibration because safe policies stop visiting violations. Its fix is to train a controlled "reckless" dual policy. We run that dual policy only in simulation, under randomization, so the critic learns the shape of the boundary. Real data then only corrects the critic's level and never requires crossing the boundary on hardware. Boundary-Seeking Policy Gradient argues that the optimal constrained policy sits exactly on the boundary and steers updates toward it. We use its update with the tightened budget $d - m_t$, so boundary seeking happens at a distance that shrinks as certification grows. Sim-to-Lab-to-Real supplies the backup shield policy and the action relabeling that keeps shield interventions from corrupting the performance policy. Its PAC-Bayes certificates need hundreds of lab environments; our calibration needs only the real rollouts that adaptation already collects.
Sim2Explore learns exploration policies in simulation and reports 6/6 real Franka puck pushes versus 0/6 for transfer-then-fine-tune. It has no safety constraints. Our gate can wrap its exploration policies directly, and one experiment does exactly that.
Research Program
The pipeline has three phases.
In simulation, a constrained policy and an $H$-step cost critic are trained under domain randomization, which means training across many randomly perturbed copies of the physics. A dual reckless policy in the style of the safety-paradox paper keeps the critic supplied with near-boundary and violating transitions. A SPiDR-style pessimistic critic and a backup safe policy are trained alongside.
On first real deployment, every state-action pair is uncertified. Behavior therefore equals SPiDR's zero-shot policy with the backup shield active, and the first real segments are collected inside that conservative envelope. At this stage the method can be no less safe than SPiDR by construction.
During adaptation, each real segment adds error scores to a calibration pool. Margins are recomputed for each $(s,a)$ from kernel-weighted quantiles over that pool. The policy is then updated in a hybrid way: the simulator generates rollouts, and the gate rule above decides which actions are allowed on hardware. The policy can creep toward the boundary only along directions where real data has piled up and errors have been small. This continuous version of "shrink the mismatch region" is what we want to show works.
The analysis has two parts. The first is a coverage statement: under exchangeability of nearby calibration and test pairs, the probability that the true $H$-step cost exceeds the gated bound is at most about $\alpha$. This follows the standard split-conformal argument, localized by the kernel. The second is an honest account of where that statement breaks. The policy changes over time, so the distribution of $(s,a)$ pairs shifts between rounds. We will measure realized coverage at every round instead of assuming it. We will also test a variant that recalibrates only on data from the most recent policy, which trades sample reuse for exchangeability.
Baselines
The naive reference is domain randomization plus PPO-Lagrangian trained in simulation, then fine-tuned on the real system with the same Lagrangian method. PPO-Lagrangian is a standard constrained policy-gradient method that adjusts a penalty weight on cost automatically. It is what most practitioners would actually do. On Safety-Gymnasium velocity tasks, PPO-Lagrangian is reported to reduce violations by 98% relative to PPO at a 45% reward drop (Safety Gymnasium). On real hardware, plain randomization gave 26.1 times more violations than SPiDR on the Go1. We beat this baseline if cumulative real violations during adaptation are lower by at least an order of magnitude at equal or better final reward.
The strongest existing method is SPiDR, run in two modes. The first is zero-shot, as published, which reports 100% constraint satisfaction on both the Go1 and the race car. The second is an online extension built by us that keeps the variance penalty during real fine-tuning, with the penalty weight tuned on the same budget as our margin hyperparameters ($\alpha$, kernel bandwidth, $n_{\min}$). Our Safety-Gymnasium PointGoal2 setup uses SPiDR's MJX port. No reported PointGoal2 reward or cost numbers were found on Lacuna for SPiDR, so we will rerun it with matched seeds and compute. We beat SPiDR if we match its violation count within a small tolerance while recovering a clearly larger share of the reward gap to an oracle constrained policy trained directly in the target dynamics. On out-of-support shifts, we beat it by staying safe where it does not.
For the tabular sanity check, the comparison is Provably Safe Sim-to-Real Transfer on its 5×5 wind gridworld. That paper reports 0% unsafe iterations versus 68.5% for unconstrained transfer, and 77,958 versus 296,910 real samples against a no-simulator baseline. In this setting we only aim to come close to its sample count without its separability assumption. We do not expect to beat an exact method in its own setting.
We also run a shielded fine-tuning baseline: the Sim-to-Lab-to-Real backup shield wrapped around SAC-Lagrangian real fine-tuning, without the lab-stage certification. It isolates what plain shielding achieves on its own.
Three ablations isolate the key ingredient. In the first, the margin comes from ensemble disagreement rather than real errors. In the second, a constant margin is applied everywhere, set to the same average value our method reaches. In the third, the simulation dual policy is removed, so the critic never sees boundary data. All methods share the same simulator budget, the same number of real environment steps, the same network sizes, and an equal hyperparameter search of twenty configurations each, selected on simulation-only validation shifts.
Why not just…?
Why not just tune SPiDR's penalty weight harder? A single global weight cannot be both large in mismatched regions and small in accurate ones. More importantly, it scales a quantity, simulator disagreement, that is blind to errors outside the randomization range. The out-of-support experiment below settles this directly. If a tuned SPiDR stays safe there with comparable reward, the main claim fails.
Why not just tighten the cost budget by a constant during fine-tuning? This is a real threat. If the simulator's error is roughly uniform across states, a constant margin captures most of the benefit with no calibration machinery. The constant-margin ablation, matched to our average margin, tests this. We expect a gain only when the error is spatially concentrated, as the mismatch-region theory suggests. We will report the shifts where it is not concentrated and our method does not help.
Why not just wrap fine-tuning in a safety filter or model predictive controller? Model predictive control is a classical approach that re-plans a short trajectory at every step using a dynamics model. Such filters are only as good as their model. Safe-Control-Gym reports that model-based controllers degrade under unmodeled disturbances, and the on-ramp merging study found that a pure MPC solver had 30–50% higher jerk than the combined RL and MPC system (On-Ramp Merging). Our method keeps a filter as its fallback, so the shielded fine-tuning baseline measures exactly how much is gained by letting calibration relax the filter.
Why not just learn a residual dynamics model online and plug it into SPiDR? This is the closest competitor, and it also uses real data. The difference is that a residual model's own uncertainty is uncalibrated, while conformal margins carry a coverage target we can check. We include this baseline as an ensemble of residual models whose online disagreement replaces SPiDR's variance. If its violation rate and reward match ours, the contribution narrows to "calibration is unnecessary," and we will say so.
Proposed Experiments
Most experiments are sim-to-sim: a held-out simulator stands in for reality, so we know the true dynamics. We use three suites. In Safety-Gymnasium PointGoal2 via the MJX port, the "real" system has shifted friction, mass, and actuation delay. In the RWRL Cartpole and Walker tasks, we perturb the dynamics. In the Safe-Control-Gym quadrotor tracking tasks, we inject step and inertial disturbances, using the Safe-Control-Gym benchmark suite (Safe-Control-Gym). Each suite has two shift families: in-support, inside the randomization range, and out-of-support, at 1.5 to 2 times the range edge. The Franka puck-pushing setup from Sim2Explore is run with an added joint-torque and workspace cost, gating its exploration policies.
The metrics are cumulative real constraint violations during adaptation, final return and cost, the fraction of the oracle's reward recovered, real samples needed to reach 90% of oracle reward, and realized conformal coverage per round. If the sim-to-sim results hold, a hardware check on a 1/10-scale race car mirrors SPiDR's platform.
Three checks are decisive. First, under out-of-support shifts, our violations stay near the $\alpha$ level while both SPiDR modes exceed it. Second, our method beats the constant-margin ablation on shifts with spatially concentrated error, and ties it on uniform shifts. Third, realized coverage stays within a few points of $1-\alpha$ despite the policy changing between rounds. If the third check fails, the margin is not doing what it claims, and the recent-policy-only recalibration variant becomes the method.
References
- Provably Safe Sim-to-Real Transfer
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Combining Reinforcement Learning with Model Predictive Control for On-Ramp Merging
- Safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics