A robot that already has a working controller (a hand-tuned safe mode, a legacy heuristic, an older learned policy) is the usual starting point for reinforcement learning on real hardware. Practitioners want to beat that controller using a simulator, without ever doing worse than it on the real machine. Two research lines each solve half of this. Baseline bootstrapping guarantees "never worse than the existing controller" by copying that controller wherever logged data is too thin to trust. Simulation, meanwhile, supplies effectively unlimited data, but that data is wrong in places. This note proposes changing what triggers the fallback. The rule should not be "copy the existing controller where real data is scarce." It should be "copy the existing controller where the simulator has not been shown to match reality." Elsewhere the policy is optimized freely in simulation, where it may crash as often as it likes.
Two consequences follow. First, the real robot never has to visit dangerous states to learn where danger lies, because the simulator explores the unsafe boundary and the real robot only has to confirm the simulator's fidelity. Second, the real data needed for that confirmation can be collected by the existing safe controller itself, followed by small, certified deviations from it. If this works, safe sim-to-real improvement becomes a loop: certify fidelity, improve inside the certified region, deploy, and gather data that grows the region. That loop has so far been shown only on a 5×5 gridworld.
Core Claim
Fix a baseline policy $\pi_b$ that is safe on the real system, a simulator with dynamics $f_{\text{sim}}$, and a small log of real transitions from $\pi_b$. We fit a model of the residual $f_{\text{real}}(s,a) - f_{\text{sim}}(s,a)$ and wrap it with split conformal prediction. Conformal prediction is a distribution-free procedure that turns any predictor into an interval guaranteed to contain the truth with probability $1-\delta$ on held-out data from the same distribution. The result is an upper bound $\hat\Delta(s,a)$ on simulator error. A state-action pair is certified if $\hat\Delta(s,a) \le \tau$ and it lies inside the support of the real data, which we check with a density gate. The deployed policy may choose any certified action. At states where it would choose an uncertified one, it executes $\pi_b$'s action instead.
We train the policy as a constrained problem entirely in simulation, with constraint costs, and the fallback is built into the simulated rollouts. Training therefore accounts for when control will be handed back. The guarantee we aim to prove takes the form
$$J^{\text{real}}_r(\pi) \;\ge\; J^{\text{real}}_r(\pi_b) + \widehat{A}_{\text{sim}}(\pi,\pi_b) - \frac{C\,\gamma\,\tau}{(1-\gamma)^2} - \xi(\delta),$$
and the same form holds for the cost return with the inequality reversed. Here $J^{\text{real}}_r$ is expected discounted real-world return, $\widehat{A}_{\text{sim}}$ is the simulated advantage of $\pi$ over $\pi_b$ accumulated only on certified pairs, $\gamma$ is the discount, $C$ is a Lipschitz constant of the value function, and $\xi(\delta)$ accounts for conformal miscoverage. The $\tau/(1-\gamma)^2$ shape is inherited from the standard simulation lemma: a one-step model error of $\tau$ compounds over an effective horizon of $1/(1-\gamma)$, and that compounded error is then summed over the same horizon.
The honest caveat is that conformal coverage holds under the baseline's state distribution. The density gate exists because the improved policy shifts that distribution. Making the guarantee survive this shift is the main theoretical risk.
Relation to Prior Work
Three lines supply the pieces. The first is baseline bootstrapping, which supplies the fallback mechanism. Safe Policy Improvement with Baseline Bootstrapping and its soft variant Safe Policy Improvement with Soft Baseline Bootstrapping copy the baseline wherever a state-action count falls below a threshold. The soft variant also notes that continuous pseudo-counts lose formal guarantees. Decision-Point Guided Safe Policy Improvement tightens the deferral bounds. All of them treat the dataset as fixed and the deployment environment as the one that generated it. We keep their deferral rule but change its trigger from data scarcity to unverified simulator fidelity, and we replace pseudo-counts with conformal bounds that carry finite-sample guarantees.
The second line is safe offline learning relative to a reference policy, which provides our strongest competitor. Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning (WSAC) penalizes the learner only when it becomes riskier than the reference. Adversarial Model for Offline Reinforcement Learning (ARMOR) frames the same relative pessimism as a game against an adversarial model. Neither uses a simulator, so neither can improve where real data is absent but simulated data is trustworthy.
The third line is sim-to-real grounding, which motivates the problem. Reinforced Grounded Action Transformation for Sim-to-Real Transfer corrects the simulator using real rollouts, but collects them with no safety constraint or fallback.
The closest prior work is Provably Safe Sim-to-Real Transfer. It certifies where the simulator matches reality in a tabular constrained problem. It explores with a state-independent mixture $(1-\alpha)\pi_b + \alpha\pi_{\text{target}}$, reports 0% unsafe iterations, and needs 77,958 real samples versus 296,910 without a simulator. It depends on counts, linear programs, and a "separable shift" assumption: the simulator is either exactly right or wrong by at least a known gap. That assumption fails for smooth physical errors. Our proposal differs in three concrete ways. The fallback is per-state, not a global blend. Certification uses a tolerance $\tau$ on a continuous residual rather than exact equality. The whole pipeline runs on continuous control.
Two works constrain the design. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safe agents starve themselves of the boundary data needed to locate constraints. Our answer is that boundary data comes from simulation, while real data is needed only for fidelity, which can be measured on the safe side. Blind Spot Detection for Safe Sim-to-Real Transfer shows that simulators can omit real phenomena entirely, and it handles this with a human queried at execution time. Our gate is meant to catch such omissions autonomously, through large residuals or through falling outside the real-data support.
Research Program
The work has three parts. The first is the certification module: a residual dynamics ensemble trained on real logs, a conformal calibration split, and a density model over (state, action) that rejects pairs far from real support. The second is the constrained policy optimizer that trains in simulation with the fallback included. For this we reuse FOCOPS-style first-order updates from Yiming Zhang 0010 et al., available in SafePO. Because handing control to $\pi_b$ is a discrete switch, we will check whether it causes the gradient-flattening that Safe Reinforcement Learning using Action Projection documents for projection layers. If it does, we will adopt their penalty-critic fix. The third is the iteration loop: deploy the hybrid policy, add its transitions to the calibration pool, recompute the certified set, and retrain.
On the theory side, the target is the inequality above, together with a bound on how fast the certified region grows per round.
Baselines
The naive reference is the baseline controller $\pi_b$ run alone, and it sets the floor. Any method whose real return falls below it in more than $\delta$ of seeds has failed the core requirement.
The first transfer baseline is zero-shot domain randomization. A PPO-Lagrangian policy is trained in a simulator whose mass, friction, and latency are randomized around the nominal values, then deployed unchanged. It gets the same simulator compute budget as our method and zero real samples. We would beat it if we achieve higher real return at equal or lower cost, or if it violates constraints under the unmodeled disturbance and we do not.
The strongest competitor that uses no simulator is WSAC, trained on the same real logs, with the same network size and tuning budget, and with $\pi_b$ as its reference. On matched-dynamics benchmarks, WSAC reports a normalized reward of 0.68 on CarCircle against 0.57 for the safe reference, and a normalized cost of 0.27 on BallCircle against a limit of 1.0 (WSAC). For constrained offline RL generally, DRCORL reports 0.91 normalized return on CarGoal1 against 0.38 for the next safe method. We will rerun DRCORL from its public code. These numbers come from settings without a simulator gap, so we report our own reruns on the mismatched versions rather than comparing to published figures. To count as beating WSAC, our method must improve on $\pi_b$ by more than WSAC does from the same real data, without exceeding the cost limit.
The strongest simulator-based competitor is grounding plus constrained training. RGAT is fitted on the same safe baseline logs, and PPO-Lagrangian is then trained in the grounded simulator and deployed. This is also the most dangerous alternative, discussed below.
The tabular mixture-policy method of Provably Safe Sim-to-Real Transfer is run on its own windy gridworld as a sanity reproduction. On that benchmark, its unconstrained comparison violated constraints in 68.5% of iterations. We also run a continuous analogue with a state-independent $\alpha$, tuned by grid search on the same real-sample budget.
Two ablations isolate the key ingredient. The first gates fallback on real-data density alone, which is SPIBB with continuous pseudo-counts and ignores the simulator residual. The second gates on the residual alone, without the support check.
Why not just…?
Why not just randomize the simulator broadly? Randomization over modeled parameters cannot cover a phenomenon the simulator lacks, such as the wind near a wall in Provably Safe Sim-to-Real Transfer or the copper-pipe wind shear in Blind Spot Detection. Wide randomization also makes controllers sluggish, as RGAT observes. Our experiments include one setting with only parametric mismatch, where randomization may well match us. We will report that result plainly. The claim being tested is that our method wins when the mismatch includes something the simulator does not model.
Why not just run constrained RL with a Lagrange multiplier on the real system? These methods satisfy constraints only at convergence. LyEvO reports SAC-Lagrangian violation rates as high as 54% in some scenarios. We will count cumulative real-world cost during learning for this baseline to make the comparison explicit.
Why not just ground the simulator on safe baseline data and train inside it? This is a real threat. If the action transformer extrapolates well beyond the baseline's support, the resulting policy could match ours in return and need no fallback machinery. Our hypothesis is that it extrapolates confidently and wrongly precisely where the improved policy goes. The decisive experiment places the mismatch just outside the baseline's typical states. Grounding-plus-training should then incur real violations there, while our gate defers in those states and certifies them only after later rounds of real data. If grounding shows no such violations, then our fallback's contribution reduces to the density ablation, and the paper's contribution narrows accordingly.
Proposed Experiments
The primary testbed is Safe-Control-Gym. The nominal CasADi Cart-Pole and 2D quadrotor models serve as "simulator." Versions with inertial-parameter error, input disturbance, and a localized step force near the constraint boundary serve as "reality." The baseline $\pi_b$ is the suite's model-based controller tuned to be safe on the perturbed system. The second testbed is modified Safety-Gymnasium/DSRL tasks, specifically CarCircle, BallCircle, and PointGoal1. Following the surrogate-reality practice of RGAT, the "real" version alters friction and actuator gain and adds an unmodeled force field near hazards. Finally, the windy 5×5 gridworld checks that we recover the tabular results.
Metrics are real return relative to $\pi_b$, normalized cost against the limit, cumulative real cost during learning, number of real transitions, and the 5th-percentile return over 20 seeds. That last metric is the analogue of the 1%-CVaR used in the SPIBB papers.
Three checks decide the outcome. The first is whether our method improves on $\pi_b$ in at least as many seeds as WSAC while keeping per-seed regressions below $\delta$. The second is whether, under the off-support hazard, the residual-only ablation and grounding-plus-training incur real violations that the full method avoids. The third is whether the certified region, and with it the margin over $\pi_b$, grows over deployment rounds rather than stalling at the baseline's footprint, which would be the continuous version of the safety paradox. A plateau would show that baseline-collected data cannot certify enough of the space, and that some deliberate, budgeted deviation is required.
References
- Safe Policy Improvement with Baseline Bootstrapping
- Safe Policy Improvement with Soft Baseline Bootstrapping
- Decision-Point Guided Safe Policy Improvement
- Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
- Adversarial Model for Offline Reinforcement Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Blind Spot Detection for Safe Sim-to-Real Transfer
- First Order Constrained Optimization in Policy Space
- Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
- Don’t Trade Off Safety: Diffusion Regularization for Constrained Offline RL
- LyEvO: Lyapunov-Guided Evolutionary Optimization for Safe and Robust Sim-to-Real Policy Learning
- Safe-Control-Gym: A Unified Benchmark Suite for Safe Learning-Based Control and Reinforcement Learning in Robotics