A robot policy trained in simulation usually has to be adapted on hardware, because the simulator gets friction, motor delays and contacts slightly wrong. The standard way to make that adaptation respect a safety budget is the Lagrangian method. A penalty weight on safety cost, the Lagrange multiplier, rises whenever the robot exceeds its budget and falls when it stays under. The flaw for hardware is structural. The multiplier only rises after the budget has already been exceeded, so on a real robot the method learns to be safe by first being unsafe. This proposal changes what drives the multiplier. Instead of reacting to violations observed on the robot, it reacts to a pessimistic forecast of real-world cost. The forecast is the simulator's cost estimate, plus a correction learned from the robot's own safe trajectories, plus a confidence margin that shrinks as real data accumulates.
All policy optimization, including any exploration near the safety boundary, happens in the simulator. The real robot only runs policies whose forecast real cost is within budget. The bet is that the reality gap in cost can be measured on safe trajectories, before it causes a violation. Cost signals such as proximity to an obstacle, contact force, or joint-limit margin typically show the simulator's optimism well before the boundary is crossed. If that bet holds, the dual variable can do its job of automatically balancing reward and safety during real-world adaptation without needing violations to learn from.
Core Claim
The problem is a constrained Markov decision process: a reinforcement learning problem where the policy $\pi$ maximizes expected return $J_R(\pi)$ subject to expected cumulative cost $J_C(\pi) \le \alpha$. Let $J_C^{sim}(\pi)$ be the policy's cost estimated by rollouts in the simulator. Let $\delta_\psi(s,a)$ be a learned correction. It predicts how much larger the real cost-to-go is than the simulator's at state $s$ and action $a$, and it is fit on real trajectories by regressing realized real cost-to-go minus the simulator critic's prediction. The proposal forecasts real cost as
$$\hat J_C^{real}(\pi) = J_C^{sim}(\pi) + \mathbb{E}_{\pi}[\delta_\psi(s,a)] + \beta_n(\pi),$$
where $\beta_n(\pi)$ is a confidence width from $n$ real episodes. It is built from the spread of residuals and inflated by a distribution-shift term when $\pi$ visits states far from the real data. The width shrinks roughly as $1/\sqrt{n}$, as a concentration bound on a mean of bounded residuals would give. The dual update then uses the forecast instead of observed real cost:
$$\lambda \leftarrow \max\big(0,\ \lambda + \eta\,(\hat J_C^{real}(\pi) - \alpha)\big),$$
with step size $\eta$. The policy is trained in the simulator on the Lagrangian $J_R - \lambda(J_C^{sim} + \mathbb{E}_\pi[\delta_\psi] - \alpha)$. A new policy is sent to hardware only if its forecast is at or below $\alpha$; otherwise the last certified policy keeps running.
The claim has three parts. First, this keeps cumulative real-world violations during adaptation near zero, even when the real dynamics lie outside the randomized range. Second, it recovers more task reward than a fixed safety margin, because the margin is earned from data rather than guessed. Third, the gain comes specifically from feeding the correction into the dual variable, rather than from the correction alone.
Relation to Prior Work
Three papers motivate and bound the design. SPiDR gets zero-shot real-world safety by adding ensemble disagreement across randomized simulators to the cost, scaled by a fixed hand-tuned weight. It does not adapt after deployment, and its penalty is only meaningful if reality lies inside the randomization range. We replace the simulator-internal uncertainty with a correction measured against reality, and let a dual variable rather than a constant set its effect. Provably Safe Sim-to-Real Transfer is the closest in spirit. It certifies where a tabular simulator is accurate, patches mismatched states with real samples, and solves a linear program. Its zero violations rely on a known safe baseline policy, a mixture weight, and the assumption that mismatches are either zero or at least a detectable size. We target continuous control with neural policies, where mismatch is everywhere and small, and we use online primal-dual updates instead of an offline linear program. Feasible Dual Policy Iteration shows the "safety paradox": safe policies stop generating violation data, so the safety estimate decays. It fixes this with a boundary-probing second policy, tested only in simulation. We borrow the diagnosis but move the probing to where it is free, the simulator, while the real robot supplies only safe data for the correction.
The method rests on primal-dual theory. Constrained Reinforcement Learning Has Zero Duality Gap and NPG-PD justify Lagrangian optimization for these problems. They also make plain its limitation of asymptotic-only feasibility, with violation shrinking only as $O(1/T^{1/4})$ under function approximation. That slow rate is exactly what a real robot cannot afford. Sim-to-Lab-to-Real enforces safety with a reachability-based shield and a staging lab. It is a complementary safety layer rather than a balancing mechanism, and we treat it as a baseline. Sim2Explore argues that coarse simulators suffice for exploration even when they are wrong for task precision. That asymmetry is why we expect simulator-side exploration plus a cost correction to be enough.
Research Program
We will build the method on SAC- and PPO-Lagrangian implementations. The correction network $\delta_\psi$ takes a state-action pair and is trained only on real transitions. The distribution-shift term in $\beta_n$ comes from an ensemble of correction networks, and grows when they disagree on states the candidate policy visits in simulation. Inside the simulator, training includes an FDPI-style probing policy near the corrected boundary, because the simulator's cost critic still suffers from the safety paradox.
Analytically, we will state conditions under which the deployed policies never exceed $\alpha$ in expectation. The condition is that the confidence width covers the residual error with high probability, with the extrapolation term bounding how wrong the correction can be off the real data. We will also extend the NPG-PD argument to show that the optimality gap of the dual iterates degrades by at most the bias of the corrected cost.
The central empirical question is whether safe-trajectory residuals predict near-boundary residuals. We test this directly by measuring correction error as a function of distance to the boundary.
Baselines
The naive reference is direct transfer of a domain-randomized policy, trained with a Lagrangian method in simulation and then run unchanged. This shows how large the reality gap in cost is on each task.
The strongest zero-shot competitor is SPiDR. It reports 100% real-world constraint satisfaction on hardware, with standard domain randomization incurring 26.1x higher violations on Unitree Go1 and 6.2x on an RC car. It is the bar for safety. We run it with the same simulator, the same randomization ranges, and its ensemble size of eight or more. Its disagreement weight gets the same tuning budget as our step size $\eta$. Beating it means matching its near-zero violations on in-support shifts, and achieving strictly fewer violations or higher return on shifts outside the randomization range, where its assumption fails. No reported result was found on Lacuna for SPiDR's return under out-of-support shifts, so this comparison is new.
The strongest adaptive competitor is PPO-Lagrangian fine-tuned on the real system with the threshold tightened by a constant margin. Margins are swept, and the best one is chosen in hindsight, which gives it an advantage. Safety Gymnasium reports PPO-Lag cutting violations 98% versus PPO at a 45% reward cost, a reminder of how conservative this route can be.
We also include the reachability shield of Sim-to-Lab-to-Real, which reports an 86.7% hardware safety rate. In settings where a safe fallback exists, we include the mixture-policy scheme of Provably Safe Sim-to-Real Transfer, which reports 0% unsafe iterations versus 68.5% for unconstrained transfer on its gridworld. We also include system identification: fit simulator parameters to the real data, then retrain.
Two ablations isolate the key ingredient. One keeps the learned correction but uses it only as a cost bonus, with a fixed multiplier. The other drives the dual variable with the correction but with no confidence width. Every method gets the same budget of real episodes and simulator steps.
Why not just…?
Why not just use SPiDR's pessimistic randomization and skip online adaptation? If reality sits inside the randomization range, SPiDR may be as safe as our method and cheaper, and we expect a tie there. Its penalty, however, measures disagreement among simulators, not error relative to reality. The doubled-gravity and 30%-friction shifts used by SR² put reality outside a moderate randomization range, and that is where the comparison is decisive. If SPiDR stays safe there too, our contribution shrinks to reward recovery, and we will report it that way.
Why not just fine-tune with PPO-Lagrangian on hardware and a tighter threshold? This is the most serious threat. A well-chosen constant margin could get most of the benefit. The difference is that our margin is computed from data and contracts with real experience, while a constant margin must be tuned per task, presumably using violations. The fixed-margin sweep reveals the hindsight-best margin. If that margin matches our return and safety, the confidence machinery is unnecessary.
Why not just identify the simulator's parameters and retrain? System identification fixes mismatch that the simulator can represent. Many of the failures listed for contact, latency and backlash are not representable as parameter changes. The cost-space correction captures whatever residual remains. If identification alone closes the gap on our tasks, that shows the correction is redundant there. To make the test meaningful, we include a shift type outside the simulator's parameter family, such as an actuator delay the simulator cannot model.
Why not just shield the real robot? A shield is only as good as its safety critic, and the critic is trained in the same mismatched simulator. Shields also only veto actions; they do not tell the policy how much reward to give up. We run the shield as a baseline and also stacked with our method, to see whether the two are complementary.
Proposed Experiments
The main suite uses sim-to-"real" pairs where the "real" environment is a held-out dynamics variant. It covers Safety-Gymnasium navigation and velocity-constrained locomotion, Safety Gym PointGoal2 in MJX, and RWRL Cartpole and Walker as in SPiDR. Shifts are graded from inside the randomization range to well outside it, using friction, gravity, morphology and unmodeled delay. If resources allow, the method then runs on a 1/10-scale RC car, mirroring SPiDR's hardware.
The primary metrics are cumulative real-world constraint violations over the whole adaptation run, final real-world return, and the number of real episodes needed to reach a given return. We also report how well the forecast is calibrated, meaning how often actual cost exceeds $\hat J_C^{real}$.
Three checks decide the proposal. First, the correction error measured near the boundary must stay within $\beta_n$. If it does not, the core bet fails and the method is unsafe. Second, under out-of-support shifts, the method must incur fewer violations than SPiDR. Third, it must beat the best fixed-margin PPO-Lagrangian in return at equal violations, and the fixed-multiplier ablation must lose to it on that comparison. Together these show that driving the dual variable with the correction is what matters.
References
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Constrained Reinforcement Learning Has Zero Duality Gap
- Natural Policy Gradient Primal-Dual Method for Constrained Markov Decision Processes
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- State Revisit and Re-explore: Bridging Sim-to-Real Gaps in Offline-and-Online Reinforcement Learning with An Imperfect Simulator