A robot trained in a physics simulator usually works on real hardware only approximately. Near obstacles, joint limits and friction edges, that approximation is exactly where the real robot gets hurt. Today there are two ways to close the gap. One is to fine-tune on the real robot, which costs many trials and risks damage. The other is to make the simulator policy so cautious that it rarely needs fine-tuning, which leaves performance on the table.
We propose a third route. Keep the simulator as the model of the world, and learn on hardware only a small correction to it, called a residual (the difference between what really happens and what the simulator predicts), together with an honest estimate of how unsure that correction is. The robot starts with a deliberately oversized safety margin. It then spends its scarce real trials probing just those states where uncertainty about the correction is what currently forces the margin to be large, shrinking the margin only as the data certify it. The bet is that real-world trials scale with how much of the operating range the simulator gets wrong, not with the size of the whole range, and that this holds without a single constraint violation during learning.
Core Claim
The closest result is Provably Safe Sim-to-Real Transfer. It shows, for small discrete problems, that if the region $B$ where simulator and reality disagree can be detected statistically, then a mixture of a safe baseline and an exploratory policy can learn safely with real samples scaling in $|B|$. On a 5x5 gridworld this took 77,958 real samples, versus 296,910 for safe learning without a simulator, with 0% unsafe iterations. That work motivates ours but also constrains it. It is tabular, it is never tested on hardware, and it assumes the simulator is either exactly right or wrong by a detectable margin. Real friction, latency and actuator drift violate that assumption continuously.
Our claim is that the discrete mismatch region can be replaced by a soft one: a learned residual whose uncertainty is used twice. First, it inflates a safety filter. Second, it directs exploration toward the states whose uncertainty most limits how much the filter can be relaxed. The resulting method should keep in-training violations at zero on continuous robots. It should also reach near-oracle performance with real-sample counts that grow with the size of the mismatch region.
This differs from Safe Exploration via Policy Priors (SOOPER) in three ways. SOOPER explores optimistically for reward with a learned ensemble dynamics model and falls back to a prior policy that is assumed already safe in the real world. We do not assume the simulator policy is safe at its nominal margin, only that an inflated-margin version of it is. We model the gap relative to the simulator rather than the full dynamics. And we direct exploration by the safety certificate rather than by reward.
Research Program
The dynamics model predicts the real next state as the simulator's prediction plus a learned residual. Let $f_{\text{sim}}(s,a)$ be the simulator's next state from state $s$ under action $a$. Let $\mu(s,a)$ and $\sigma(s,a)$ be the mean and spread of an ensemble of small networks trained on real transitions to predict $s'_{\text{real}} - f_{\text{sim}}(s,a)$. The ensemble is initialized to output zero, so that with no data the model defaults to the simulator. The shield accepts an action only if every state in the inflated prediction set
$$\hat{S}(s,a) = \{\, f_{\text{sim}}(s,a) + \mu(s,a) + \delta : \|\delta\| \le \beta\,\sigma(s,a) + m \,\}$$
lies inside the region from which a backup policy can return to a known safe set. Here $\beta$ is a confidence multiplier and $m$ is an extra safety margin. The backup-return check reuses the statistical trajectory-sampling test of Safe Reinforcement Learning via Statistical Model Predictive Shielding, run through the residual-corrected model instead of a nominal one. We reuse its technique; its reported speed (0.39 s per step at 99% confidence) makes it plausible at hardware control rates.
The backup policy is the simulator-trained policy, trained with domain randomization and run with a large clearance and speed cap. The margin $m$ starts large enough that this backup is safe across the randomized envelope. We treat that as an assumption, checked in a short slow-speed hardware pilot, rather than a guarantee.
Exploration then follows a certificate-directed score. Among actions the shield accepts, the robot prefers those that most reduce residual uncertainty at the frontier of the safe region, the states where the shield's constraint is currently the binding one. Concretely, the score is $\sigma(s,a)$ weighted by how close $\hat{S}(s,a)$ comes to the backup-reachable boundary. It is mixed with the task policy's own action choice through a probability $\alpha$ that anneals as the margin shrinks. This mixing is the continuous analogue of the $\alpha$-mixture in the tabular paper.
This scoring is our direct answer to the safety paradox identified in Breaking Safety Paradox with Feasible Dual Policy Iteration: safer policies see fewer boundary samples and so learn the boundary worse. Here the robot is sent to the boundary on purpose, but only to the part of it the certified backup can recover from. After each batch of real data, $m$ is decreased when held-out real transitions fall inside the predicted sets at the target rate.
The task policy is trained in the residual-corrected simulator, so that nearly all gradient steps use simulated rollouts. Constraints are enforced as a constrained problem with a Lagrange multiplier rather than a hand-set penalty. Because reality drifts (battery sag, heating), a running coverage monitor re-inflates $m$ whenever the recent fraction of real transitions outside $\hat{S}$ exceeds the target. We do not claim exchangeability-based guarantees of the kind Conformal Prediction for Uncertainty-Aware Planning with Diffusion Dynamics Model provides offline. Its own limitation, that exchangeability fails under sim-to-real shift, is why we use an adaptive monitor instead.
Baselines
The naive reference is zero-shot transfer: the domain-randomized simulator policy deployed with no real learning, both at its nominal margin and at the inflated starting margin. This brackets the unsafe-but-capable and safe-but-slow extremes. The tabular paper reports that unconstrained sim-to-real reached 68.5% unsafe iterations on its gridworld, which indicates how badly the nominal-margin version can fail when mismatch sits near hazards.
The strongest existing method is SOOPER from Safe Exploration via Policy Priors. It reports zero violations on its simulated tasks, a hardware RC-car policy with twice the prior's reward, and no training collisions. We give it the same inflated-margin simulator policy as its prior, the same real-step budget, the same ensemble size and a matched hyperparameter search. It will be run twice. The first variant uses its ensemble trained from scratch on real data. The second pre-trains that ensemble on simulator data, which is the fairest version of the "just warm-start" argument. Beating it means an equal or lower violation count together with at least 2x fewer real steps to reach 90% of oracle return, or a significantly higher return at equal steps.
On the gridworld, the tabular method from Provably Safe Sim-to-Real Transfer is the reference. We should roughly match its 77,958 real samples and 0% unsafe iterations there; doing worse would indicate that our continuous machinery loses sample efficiency.
We also run grounding without safety: an action-transformer method as in Reinforced Grounded Action Transformation for Sim-to-Real Transfer, which reports matching real-trained performance on MuJoCo but no numeric returns in the file. It is run with and without our shield bolted on. A constrained model-free learner (PPO with a Lagrange multiplier) trained directly on the "real" system gives the sample-cost ceiling. The Safety Gymnasium paper reports a 98% violation reduction for it over plain PPO in velocity tasks, at a 45% reward cost.
Two ablations isolate the key ingredients. The first replaces certificate-directed probing with passive data collection under the backup policy, holding the residual model fixed. The second replaces the sim-plus-residual model with a full learned model at the same capacity.
Why not just…?
Why not just randomize the simulator harder and deploy zero-shot? Randomization only covers parameters someone thought to vary. Blind Spot Detection for Safe Sim-to-Real Transfer shows agents become confidently wrong when the simulator lacks a feature of reality. Wide randomization also makes policies sluggish. Our zero-shot baselines at both margins measure this directly. If the nominal-margin policy has zero violations and near-oracle return on hardware, the problem we target does not exist on that platform, and we would report that.
Why not just run SOOPER with a simulator-pretrained model? This is the real threat. A pretrained ensemble may absorb the simulator almost as well as an explicit residual, and reward-optimistic exploration may happen to visit the same boundary states. The two ablations and the pretrained-SOOPER variant settle it. If they match our real-step counts, the contribution reduces to the margin-shrinking start from an unsafe-at-nominal prior, and we would present it that way. We expect a gap mainly when mismatch is concentrated near hazards, because there reward-seeking exploration and safety-relevant exploration diverge.
Why not just collect real data passively with the cautious policy, then ground the simulator once? This is cheaper to build. But a cautious policy never visits the boundary states the final policy will operate near, so the correction stays untrained exactly where it matters. The suboptimal-oracle result in the blind-spot paper is evidence for this: classifier AUC was 0.96 with boundary-visiting data versus 0.60 with purely safe data. The passive-collection ablation tests it in our setting.
Why not just a hand-designed barrier-function filter over the simulator policy? Static filters need accurate analytic dynamics and can deadlock in front of obstacles. Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions and Safe exploration in model-based reinforcement learning using control barrier functions both report fixed controllers getting stuck behind obstacles. A filter at a fixed margin is our zero-learning baseline, so its conservatism will show up as a return gap.
Proposed Experiments
The first stage is a sanity check on the 5x5 windy-cell gridworld from the tabular paper, discretizing our method's outputs. It should reproduce the 0% unsafe rate at a sample count near the reported 77,958.
The main simulated study uses Safety Gymnasium navigation tasks (Point, Car and Racecar agents on Goal and Button). The "real" environment is created by injecting mismatch the nominal simulator lacks: a friction patch, a wind field localized near hazards, and actuation latency. The latency setting follows the delays that An empirical investigation of the challenges of real-world reinforcement learning found catastrophic for D4PG and DMPO. We repeat this on its Cartpole and Walker tasks. No reported result was found on Lacuna for any method on these injected-mismatch variants, so all baselines are run by us under the matched budget above.
Metrics are cumulative training violations, real steps to 90% of an oracle trained on the mismatched system, final return, final margin $m$, and the empirical coverage of $\hat{S}$ over time.
The decisive scaling check varies the fraction of the state space covered by mismatch (patch area from 5% to 80%) with fixed magnitude. Our claim predicts that real steps to target grow roughly in proportion to that fraction; this is the continuous version of the $|B|$ dependence proven in the tabular setting. SOOPER from scratch should stay roughly flat because it models everything. If our curve is also flat, the central claim fails.
A drift test slowly ramps the mismatch magnitude mid-training. Success means violations stay at zero, via monitor-triggered re-inflation, and do not come from freezing.
The hardware stage uses a small instrumented RC car on an obstacle course, chosen so the comparison to SOOPER's own hardware result is as direct as possible. State comes from onboard sensing rather than motion capture, which targets the reliance on external tracking that limits the Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions Crazyflie result. Mismatch comes from a low-friction floor section and added payload the simulator does not model. The budget is a fixed number of real minutes, with three seeds per method, for our method, both SOOPER variants and the passive-collection ablation.
We count collisions and near-miss events (clearance below half the final margin), time to reach twice the backup policy's lap reward, and final lap time. The result that would justify the method is zero collisions for every seed together with a clear reduction in real minutes versus simulator-pretrained SOOPER on the friction-patch course. If that reduction disappears on hardware while holding in simulation, we would conclude that the residual's uncertainty is miscalibrated under real sensor noise, and report that as the finding.
References
- Provably Safe Sim-to-Real Transfer
- Safe Exploration via Policy Priors
- Safe Reinforcement Learning via Statistical Model Predictive Shielding
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Conformal Prediction for Uncertainty-Aware Planning with Diffusion Dynamics Model
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Blind Spot Detection for Safe Sim-to-Real Transfer
- Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions
- Safe exploration in model-based reinforcement learning using control barrier functions
- An empirical investigation of the challenges of real-world reinforcement learning