Borrowed Danger: Safe Real-World PPO Fine-Tuning Without Waiting for Crashes

A robot policy trained in simulation usually meets the real world with dynamics the simulator got wrong: more friction, extra latency, a motor that responds late. The safe response is to keep learning on the real robot, but this runs into a basic problem. Safety feedback on hardware is sparse and binary. The robot either crashes or it does not. A policy that was careful in simulation will rarely crash in reality, so it gets almost no signal about where the true danger boundary has moved, until it crosses that boundary.

We propose to fix this with a signal the robot already has. The simulator can still tell the robot which states are dangerous, even when it is wrong about how the robot gets there. So for every real transition, we score the state the robot actually landed in using a danger estimate learned in simulation. That score is then carried back to the state and action that produced the transition. If the real dynamics push the robot toward trouble faster than the simulator predicted, the robot finds out one step early from a near-miss, not from a collision. These relabeled real transitions drive Proximal Policy Optimization (PPO, a policy-gradient method that limits how far each update can move the policy). PPO's update limit is tightened only in states where real transitions visibly disagree with the simulator.

Core Claim

The claim has two parts. First, the danger learned in simulation transfers much better as a function of state than as a function of action, because what the simulator usually gets wrong is the transition dynamics, not where the walls are. Second, bootstrapping a simulation-trained feasibility score through real transitions produces a dense correction signal for the real danger boundary. The feasibility score is the largest discounted future cost along a trajectory, from Feasibility Consistent Representation Learning for Safe Reinforcement Learning. Because this correction signal needs no observed real violations, it breaks the "safety paradox" of Breaking Safety Paradox with Feasible Dual Policy Iteration: safer policies see fewer violations, so their cost estimates get worse.

We predict that this lets PPO fine-tune on shifted dynamics with far fewer real violations than fine-tuning on real costs alone. We also predict it recovers more of the task return than a pessimistic zero-shot policy, and that the advantage is largest when the real dynamics fall outside the randomization range used in simulation.

Concretely, let $c(s)\in\{0,1\}$ be the binary violation indicator and $\gamma$ the discount. The feasibility score satisfies the max-Bellman relation

$$F(s,a) = \max\big(c(s),\ \gamma\, \mathbb{E}_{s'\sim P}\, F(s', \pi(s'))\big),$$

where $P$ is the transition law. In simulation, $F_{\text{sim}}$ is trained on a randomized ensemble of simulators. On hardware, we fit a real feasibility score $F_{\text{real}}$ by applying the same backup to real transitions $(s,a,s')$. It is anchored at $s'$ by $F_{\text{sim}}$ wherever $s'$ has not yet been visited enough in reality to trust $F_{\text{real}}$. A real transition that lands where $F_{\text{sim}}(s')$ is high raises $F_{\text{real}}(s,a)$ immediately, whether or not a crash ever happens. This correction is only valid when the unsafe set itself (walls, joint limits) matches between simulator and reality and only the dynamics differ. We treat that as an explicit assumption and test what happens when it breaks.

Relation to Prior Work

Three lines of work motivate the gap.

The first handles transfer without online correction. SPiDR adds a penalty for ensemble transition variance to the cost and achieves zero-shot safety on a Unitree Go1 and an RC car. It stays static after deployment and assumes reality lies inside the randomization envelope. Sim-to-Lab-to-Real replaces binary failure with continuous reachability margins and shields fine-tuning with them. It relies on accurate distance sensing and hundreds of lab environments, and it does not correct its safety critic from real dynamics.

The second handles online correction, but only in small settings. Provably Safe Sim-to-Real Transfer shows that real-world sample cost should scale with the size of the region where simulator and reality disagree. It does this only for tabular problems solved by linear programming. Our mismatch gating is a continuous, deep-network analogue of its restriction to the mismatch region, though without its guarantee.

The third handles sparse safety feedback, but only inside one simulator. SB-TRPO shows that neural cost critics fail under sparse binary violations and replaces them with Monte Carlo estimates inside a second-order trust region. EVO models rare violations with extreme-value statistics, but it needs such violations to appear in its replay buffer.

We supply the method from two sources. From FCSRL we reuse the feasibility score, which that paper proves is temporally smoother than cumulative cost. From Lagrangian PPO as implemented in Safety Gymnasium's SafePO library we reuse the constrained PPO machinery. What we add is neither a new critic nor a new trust region. It is the use of real transitions plus simulation-learned state danger as the correction signal.

One cited result pushes against part of our design. Backward value functions enforce state-wise budgets during training, but they depend on a dynamics model that sim-to-real shift corrupts. This is part of why we anchor on state-based danger rather than dynamics-based bounds.

Research Program

We build the method in three stages.

In simulation, we train a PPO-Lagrangian policy together with $F_{\text{sim}}$ across a randomized dynamics ensemble. We also train a small ensemble of one-step dynamics models.

On the "real" system, we run PPO fine-tuning with four additions. First, $F_{\text{real}}$ is fit by the anchored max-backup above. Second, a per-state mismatch score measures how far the real next state falls outside the spread of the ensemble's predictions. Third, the cost advantage is computed from $F_{\text{real}}$ rather than from the sparse real cost. Fourth, reward and feasibility advantages are clipped separately. The clip range is kept tight for any risk-increasing change in flagged high-mismatch states and at the normal PPO setting elsewhere. This responds to the concern that clipping a single combined advantage can cut off exactly the updates that restore safety.

We treat the separate clipping as secondary and will drop it if ablations show it adds nothing. We then study failure modes. The first is out-of-support shifts. The second is shifts in the unsafe set itself, where the method should degrade to plain PPO-Lagrangian and we expect it to. The third is the anchoring schedule that decides when $F_{\text{real}}$ stops deferring to $F_{\text{sim}}$.

Baselines

The naive reference fine-tunes the domain-randomized PPO-Lagrangian policy on the shifted system using only the observed binary costs, with the SafePO implementation from Safety Gymnasium. This is the setting where the safety paradox should bite. The paper reports that PPO-Lag achieved "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward" on velocity tasks. That figure is for training in simulation, not for fine-tuning after a shift. Beating this baseline means fewer cumulative real violations at equal real-sample budgets.

The strongest competitor is SPiDR. It reports "100% constraint satisfaction in the real world" across its hardware tests and "26.1x higher safety violations" for standard domain randomization on the Go1. We will run it both zero-shot and with its penalty kept on during PPO-Lagrangian fine-tuning. It gets the same ensemble size (at least 8, as in the paper), the same simulation budget, and the same hyperparameter search budget over its penalty weight as we give our anchoring and clipping settings. Beating it means matching its violation count while recovering significantly more return, or keeping safety under out-of-support shifts where it fails. Matching it inside the randomization support would be an honest null result.

We add SB-TRPO-style Monte Carlo cost estimation during real fine-tuning. It is the best existing answer to sparse costs. It reports safe reward and safety probability but gives no transfer numbers; no reported result was found on Lacuna for SB-TRPO under dynamics shift. We also add a reachability shield in the style of Sim-to-Lab-to-Real. Its hardware numbers ("86.7% safety rate") come from a different navigation task and are not directly comparable.

Ablations isolate the key ingredient. One replaces $F_{\text{real}}$ with $F_{\text{sim}}$ frozen, with no relabeling. One removes the mismatch gate. One replaces separate clipping with a globally small clip range.

Why not just…?

Why not just deploy SPiDR zero-shot and stop? If reality lies inside the randomization envelope, this may well be enough, and our experiments include such in-support shifts to check. SPiDR states that its guarantee depends on that envelope assumption. It also pays for safety with pessimism everywhere the ensemble disagrees, which is the "freezing" pathology the file documents. The decisive comparison is return and violation count under shifts placed deliberately outside the randomization range.

Why not just fine-tune with PPO-Lagrangian and a very conservative multiplier? A large multiplier still acts on a cost signal that is almost always zero. It either does nothing until a crash or, if set high, produces the "safe but useless" behavior described by Quanyi Li et al.. SB-TRPO attributes Lagrangian oscillation partly to noisy critics trained on scarce violations. Our naive baseline, swept across multiplier settings, will show whether any setting matches our violation curve.

Why not just shrink PPO's clip range globally? This is a real threat to the separate-clipping component. It slows risky changes, but it also slows learning of the task everywhere. It is our ablation. If a global small clip range matches the gated version, we drop the gating and keep only the relabeling, which the global clip range cannot replace because it adds no information about where the boundary moved.

Why not just add a shield? Static shields inherit the simulator's mistaken dynamics, as argued in Adaptive Probabilistic Shielding. Learned reachability shields need specialized training pipelines. We include a reachability shield as a baseline rather than dismissing it.

Proposed Experiments

The main testbed is sim-to-sim transfer with controlled dynamics shifts, where violations cost nothing and the true boundary is known. We use Safety Gymnasium PointGoal2 and CarButton, plus the velocity-limited locomotion tasks, ported to MJX as in SPiDR. We apply perturbations from Robust Gymnasium. Friction, actuator latency, and mass are set either inside or outside the randomization range, and there is a structured "wind" drift analogous to the windy-cell gridworld of Provably Safe Sim-to-Real Transfer. That gridworld itself serves as a sanity check: its reported "0% unsafe iterations" versus "68.5%" for the unconstrained baseline gives a target our deep method should come close to.

The metrics are cumulative violations during fine-tuning, final return, and SB-TRPO's safe reward and safety probability. We also track the number of real samples needed to recover 90% of in-simulation return. All results use at least five seeds.

The decisive check is a plot of violations against real samples for out-of-support shifts. If relabeling works, our violation curve should rise mostly before the first few hundred episodes and then flatten. The real-cost baseline should instead keep accumulating violations as it slowly discovers the moved boundary. A second check tracks how often the first danger signal comes from a feasibility-score spike rather than from an actual violation. This directly measures whether the danger borrowed from simulation arrives early.

A negative control shifts the unsafe set itself, for example by moving hazards the simulator does not model. Here we expect no advantage over PPO-Lagrangian. A win there would indicate our gains come from something other than the stated mechanism.

If the simulation results hold, a small hardware study on a 1/10-scale RC car with added payload and latency would follow the setup of SPiDR's RaceCar experiments.