A robot trained in a simulator and moved to the real world faces a choice that current methods make badly. It can stay cautious forever, avoiding anything the simulator is unsure about. That is safe, but it never recovers the performance the reality gap cost it. Or it can keep learning on the real hardware, which means brushing against real safety limits, sometimes crashing, to find out where those limits actually are. This proposal argues that the two jobs can be separated. The simulator hosts the risky job of deliberately probing where failure begins. The real robot does only the cautious job of checking, from ordinary safe operation, which parts of the simulator can be trusted. Caution is then relaxed exactly where reality has confirmed the simulator, and nowhere else.
The payoff we aim for is a specific kind of convergence guarantee. Most learning guarantees measure total loss over time, and they allow an agent to make serious mistakes forever as long as the mistakes grow rare. We instead count, for every tolerance level at once, how many real-world episodes the robot spends either unsafe or meaningfully below the best safe behavior. The target is that this count stays bounded for every tolerance. A permanently cautious transfer policy fails this test at small tolerances: it is never unsafe, but it stays suboptimal in every episode. An online safe learner fails it through unsafe episodes during training. We want a method that passes on both counts, and an evaluation protocol that makes the difference visible in continuous-control benchmarks.
Core Claim
Our claim is that a sim-to-real agent can have zero real-world constraint violations (with high probability, given a known safe fallback policy) and a bounded number of real-world ε-mistakes for every ε simultaneously. Three things are needed. First, the pessimism penalty that keeps the transferred policy safe must shrink according to time-uniform confidence widths of a learned sim-to-real residual model, not according to a hand-set constant. Second, the feasibility estimate, meaning the learned predictor of whether a state can still avoid failure, must be kept accurate near the boundary by running an adversarial "boundary-seeking" policy only inside the simulator after it has been corrected with real data. Third, real-world exploration must be mixed with the safe fallback in proportion to the remaining uncertainty.
Formally, let $\pi_k$ be the policy deployed in real episode $k$, $V^*$ the value of the best policy that satisfies the constraint in the real system, $V^{\pi_k}$ its real value, and $J_c(\pi_k)$ its real expected cost against budget $b$. The quantity we bound is
$$N_T(\epsilon) = \#\{k \le T : V^* - V^{\pi_k} > \epsilon \ \text{or}\ J_c(\pi_k) > b\},$$
and the goal is $N_T(\epsilon) \le F(\epsilon)$ for all $\epsilon > 0$ and all $T$, with the unsafe part of the count equal to zero. This is the Uniform-PAC notion of Dann et al. (2017), measured on real episodes only, with safety folded in.
Why this is the open spot
Each direction has solved part of the problem while leaving the neighboring part open. On the convergence side, Zhou et al. (2021) point out that even logarithmic regret allows severe errors infinitely often at exponentially spaced times. Uniform-PAC closes that loophole, and UOpt-RPGPD extends it to constrained problems with last-iterate convergence. However, it learns from scratch in one environment, permits violations early in training, and has only been tested on a 30-state synthetic problem. Cert-LSVI-UCB reaches constant regret under bounded model misspecification, which is suggestive for sim-to-real because a simulator is a misspecified model. It has no notion of constraint cost, though, and its bound blows up as the suboptimality gap shrinks. These papers supply our target and our tools: time-uniform confidence bounds and certified pessimistic/optimistic crossing tests. They do not supply a simulator or a zero-violation guarantee.
On the transfer side, SPiDR adds ensemble disagreement across randomized simulators to the cost and reports 100% constraint satisfaction on a Unitree Go1 and RC cars. It never updates online, and its penalty weight is fixed by hand. In our terms it has zero unsafe episodes but an $N_T(\epsilon)$ that grows linearly for any ε below its conservatism gap. Provably Safe Sim-to-Real Transfer is the closest work. It mixes a safe baseline with an explorer to get zero real violations, and shows that real sample cost scales with the mismatch set rather than the whole space. But it is reward-free, tabular, and relies on an assumption that each state-action either matches the simulator exactly or differs by at least a fixed amount. We differ from it in three ways: we are task-directed and measure convergence of the deployed policy, we replace the separability assumption with shrinking confidence widths that tolerate small, diffuse mismatch, and we target continuous control.
The third ingredient comes from FDPI. It shows that as an agent becomes safer, failure data vanishes from the replay buffer, the feasibility estimate degrades, and late cost spikes follow. Its fix is a KL-tethered adversarial policy that seeks violations, which cannot run on hardware. This result both motivates and constrains our design. Any real-world learner that becomes safe will lose boundary data, so boundary probing is necessary. And the probing cannot happen in reality. Our move is to host FDPI's adversarial policy in the corrected simulator. This carries a real risk: simulator bias is likely largest precisely near the boundary. That risk is the central empirical question of the proposal.
Research Program
Mechanism. We start from a simulator (possibly domain-randomized) and a known safe fallback policy $\pi_{safe}$, for example a SPiDR policy with large penalty. On real data we fit an ensemble residual model predicting the difference between the real next state and the simulated next state. Its disagreement gives a width $w_n(s,a)$ that shrinks as real visits to $(s,a)$ accumulate. Unlike SPiDR's sim-only variance, this width reflects what reality has actually shown. The policy is optimized in the corrected simulator (simulator plus residual) against the cost
$$\tilde c_n(s,a) = c(s,a) + \beta_n\, w_n(s,a),$$
where $\beta_n$ grows like the time-uniform $\sqrt{\ln \ln n}$ confidence scaling used by UBEV and UOpt-RPGPD rather than being a tuned constant. This is what should prevent the agent from settling forever into an ε-suboptimal cautious policy.
Inside the corrected simulator only, an FDPI-style dual policy seeks violations near the current primal policy's support, with importance weighting. This keeps the feasibility critic trained on dense boundary data at zero hardware risk. On the real system, the executed policy is a mixture of the optimized policy and $\pi_{safe}$. The mixture weight is set so that the pessimistic cost bound stays below budget, following the construction of Provably Safe Sim-to-Real Transfer. A Cert-LSVI-UCB-style crossing test flags regions where the optimistic and pessimistic cost estimates disagree beyond what the widths allow. Those regions have their widths reset, which gives protection when the residual model itself is wrong.
Theory. The theory is deliberately confined to where proofs are realistic. For tabular CMDPs, we aim to show that the ε-mistake count on real episodes is bounded uniformly in ε, with zero violations with high probability. The bound should depend on the real data needed to shrink widths along trajectories of near-optimal safe policies, not on $|S \times A|$, and it should not use σ-separability. The natural gap-dependent form is that mismatches too small to change the ranking of near-optimal safe policies never need to be resolved. Under a positive suboptimality gap we will also test whether action elimination yields constant regret, extending Cert-LSVI-UCB to constraints. If only the tabular Uniform-PAC statement is proved, the proposal still stands. Continuous control is an empirical claim, and we say so.
Baselines
The naive reference point is zero-shot domain randomization deployed and frozen. We will run it with the same simulator, randomization ranges, and policy-optimization budget as every other method. On hardware, SPiDR reports 26.1x more safety violations for this approach than for SPiDR on the Go1 and 6.2x more on the RC car. We expect it to fail on safety whenever the boundary moves inward.
The strongest safety baseline is SPiDR itself, frozen. We sweep its penalty weight over the same tuning budget we give our β schedule, and pick the setting that is safe on a held-out perturbation. Its reported result is 100% constraint satisfaction on real hardware. For the RWRL and PointGoal2-MJX environments we reuse, there is no reported result found on Lacuna for SPiDR's simulation numbers, so we will reproduce it. Beating SPiDR means matching its zero violations while its $N_T(\epsilon)$ grows linearly at small ε and ours plateaus, visible as final real return approaching the real constrained optimum.
The strongest adaptive baseline is SPiDR followed by real-world safe fine-tuning using PPO-Lagrangian with PID dual updates from SafePO (Safety Gymnasium), gated by a reachability shield in the style of Sim-to-Lab-to-Real. That paper reports an 86.7% hardware safety rate. It gets the same number of real episodes as our method. Beating it means fewer unsafe real episodes at equal or better final return.
A grounding baseline, RGAT plus SPiDR, grounds the simulator with the action transformer of RGAT using fallback-policy data, then retrains with SPiDR pessimism. It is given the same real data.
For the tabular check we compare against the published method on its 5x5 windy gridworld. That method reports 0% unsafe iterations and 77,958 real samples, versus 296,910 without a simulator. We also compare against UOpt-RPGPD run from scratch.
Three ablations isolate the ingredients. The first replaces the data-dependent width with a global λ annealed on a fixed schedule. The second removes the simulator-hosted dual policy and keeps ordinary replay. The third runs the dual policy in the uncorrected rather than the corrected simulator.
Why not just…?
Why not just use SPiDR with a well-tuned penalty? It is the right answer if the only goal is safety, and it will likely tie us on violations. The problem is that any fixed penalty that is safe under an inward boundary shift is too cautious under an outward one, and it can never learn which case it is in. Our outward-shift environments, where reality is more forgiving than the pessimistic ensemble, measure exactly this. If SPiDR's return there is already within noise of the real constrained optimum, our contribution shrinks to the inward-shift case, and we will report that.
Why not just anneal the penalty on a schedule during real fine-tuning? This is the most serious threat, because it is simple and might recover most of the return. A global schedule cannot tell the regions reality has confirmed from the ones it has not. It will either relax too early in unvisited regions near the boundary, causing violations, or too late everywhere. The first ablation tests this directly. If global annealing matches our violations and ε-mistake curves across the shift suite, the width-based design is not justified and we will simplify to it.
Why not just ground the simulator with RGAT and retrain? Grounding corrects the average dynamics but says nothing about where the correction is trustworthy, and RGAT collects its real data without constraints. Our version collects grounding data only through the safe mixture, so the comparison isolates whether confidence widths and boundary probing add anything beyond a better simulator.
Why not just up-weight rare failures with prioritized replay? FDPI examined passive re-weighting and found that it cannot replace fresh near-boundary samples once they stop arriving. Our second ablation re-tests this under transfer, where the stale failures come from the wrong dynamics.
Proposed Experiments
The main suite is sim-to-sim, because we need the "real" optimum to compute $N_T(\epsilon)$ exactly. In Safety-Gymnasium PointGoal, CarGoal, and the velocity-constrained locomotion tasks, and in the RWRL Cartpole and Walker and the MJX PointGoal2 port used by SPiDR, the "real" system is a perturbed copy. Perturbations include friction, mass, actuator latency, and lateral pushes, drawn partly from RRLS. We split perturbations into inward shifts, which move the constraint boundary toward nominal behavior (for example, lower friction lengthens braking distance), and outward shifts, which make nominal caution unnecessary. Some perturbations fall outside the randomization support, to probe SPiDR's coverage assumption. The real optimum is estimated by long-budget constrained training directly on the perturbed system, and we check it for optimality rather than mere stability, following the warning of Lim et al. (2025) that convergent RL can settle far from optimal.
Metrics are the number of unsafe real episodes, the curve $N_T(\epsilon)$ over a grid of ε, final real return relative to the optimum, and real samples used. The tabular theory is checked on the 5x5 windy gridworld and the 30-state CMDP, adding reward to the former. RC-car hardware is a follow-up, contingent on the simulation results.
Three outcomes would decide the proposal. The first concerns inward shifts: if the dual policy hosted in the corrected simulator keeps the feasibility error near zero and removes late cost spikes, the core idea holds. If it instead learns a biased boundary and real violations appear, then simulator bias near the boundary defeats the design. In that case, the third ablation, which runs the dual policy in the uncorrected simulator, tells us whether correcting the simulator is the fix. The second concerns outward shifts: our $N_T(\epsilon)$ must flatten at small ε where frozen SPiDR's grows linearly, which shows the convergence gap is real and closable. The third concerns the global-annealing ablation: it must lose on one of these two axes, or the local-width machinery is unnecessary.
References
- Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning
- Uniform-PAC Bounds for Reinforcement Learning with Linear Function Approximation
- A Policy Gradient Primal-Dual Algorithm for Constrained MDPs with Uniform PAC Guarantees
- Achieving Constant Regret in Linear Markov Decision Processes
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration