Robots trained in simulation to respect a safety budget, such as "expected collisions per episode below a limit," usually exceed that budget once deployed, because real physics differs from the simulator. The natural fix is to keep training on the robot with the same constrained-RL machinery used in simulation. That machinery keeps a penalty weight on safety cost (the Lagrange multiplier) and nudges it up or down based on how far the measured cost is from the budget. It has a known failure mode called the "Lagrangian seesaw." The penalty overshoots, the policy becomes timid, then the penalty decays and the policy becomes reckless again. PID Lagrangian methods damp this by treating the penalty update as a feedback controller, adding proportional and derivative terms on top of the usual running sum of errors.
On hardware those extra terms run into a problem the simulation papers never faced. Each update gets only a handful of real episodes, so the measured cost is very noisy, and the derivative and proportional terms amplify that noise. The usual remedy is a smoothing filter, but filtering adds lag, and lag is exactly what causes the seesaw. We propose to cut the noise without adding lag. Every real episode is replayed in the simulator from the same initial state, and the simulator's own cost is used as a control variate: a correlated, cheaply measured quantity whose known average lets us subtract out the shared randomness. The simulator stops being only a pre-training tool and becomes a low-noise sensor inside the safety controller during real-world fine-tuning.
Core Claim
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods showed that the standard multiplier update is a pure integral controller. It also showed that adding a proportional term damps cost oscillations in Safety Gym. It flagged the noisiness of the derivative term on stochastic cost estimates as an open problem and evaluated only in simulation. Our claim is that on a real system this noise problem dominates, and it has a structural solution unavailable in pure simulation. Much of the episode-to-episode variation in real cost comes from things the simulator reproduces: start state, obstacle layout, and the policy's own sampled actions. That shared variation can be cancelled.
Concretely, at dual update $t$ let $\bar C_{\text{real}}$ be the mean cost over $n$ real episodes. Let $\bar C_{\text{pair}}$ be the mean cost of the same policy in simulation from those same $n$ initial states. Let $\bar C_{\text{sim}}$ be the simulator cost averaged over $N \gg n$ fresh episodes. The error fed to the PID controller is
$$\hat e_t = \bar C_{\text{real}} - \beta\left(\bar C_{\text{pair}} - \bar C_{\text{sim}}\right) - V_0,$$
where $V_0$ is the safety budget and $\beta$ is a coefficient fit online by regressing real on paired costs. This estimator is unbiased for real cost minus budget for any fixed $\beta$. At the optimal $\beta$ its variance is roughly $(1-\rho^2)$ times that of $\bar C_{\text{real}}$, where $\rho$ is the per-episode correlation between real and paired-sim cost. This is the standard control-variate result, with $N$ assumed large. When the simulator is good, $\rho$ is high and the proportional and derivative terms become usable on a few episodes per update with no filter lag. When the simulator is bad, the fitted $\beta$ shrinks toward zero and the method falls back to ordinary PID Lagrangian. The reality gap itself, the systematic offset between real and simulated cost, still reaches the controller in full through $\bar C_{\text{real}}$, so the integral term corrects it.
Research Program
The work has three parts.
Estimator and fallback. We first build the paired estimator into a PID-Lagrangian Proximal Policy Optimization learner, reusing the reward-scale normalization of the PID Lagrangian paper. We then characterize empirically how $\rho$ varies with the size of the dynamics mismatch, and confirm that the fallback to plain PID behaves as predicted when $\rho$ collapses.
Gain tuning under the hardware data budget. The second part addresses gain tuning, which the PID paper leaves manual. Gains tuned in simulation, where each dual update sees hundreds of episodes, are mis-scaled for hardware, where each update sees a few. We will tune the gains in simulation under an emulated hardware budget: the same $n$ episodes per update, the same paired-replay estimator, and injected actuation latency. We then ask whether these gains transfer to the "real" side without retuning. This turns the tuning question into a concrete, testable protocol rather than a per-robot search.
Where the cancellation works. The third part is diagnostic. We will measure when the cancellation helps and when it fails. Failure cases include unmodeled latency that desynchronizes paired episodes, contact-rich failures that the simulator mispredicts, and state-reset error when initial states can only be estimated.
We deliberately do not add a shield or a pessimism bonus. Those address a different question (see below) and can be layered on afterward.
Baselines
The naive reference point is zero-shot deployment of a domain-randomized constrained policy with no real-world updates. SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer reports that plain domain randomization produced 26.1x more violations than SPiDR on the Unitree Go1 and 6.2x more on a 1/10-scale race car. This fixes the scale of the problem.
The strongest zero-shot competitor is SPiDR itself, which adds ensemble disagreement as a pessimistic cost and reports 100% constraint satisfaction on both robots. We run it with the same simulator, domain-randomization ranges, and training compute as our pre-training stage.
The strongest fine-tuning competitor is PID Lagrangian (Stooke et al.) warm-started from the simulated multiplier. It uses the same policy, the same real-episode budget per update, and gains tuned with the same number of tuning trials as ours. It is evaluated both with its raw cost estimate and with an exponential moving-average filter whose time constant is swept. We also include integral-only Lagrangian fine-tuning, and the inverse-multiplier learning-rate variant from Adaptive Primal-Dual Method for Safe Reinforcement Learning as a different route to stability. For neither of these does the file contain reported numbers on our target tasks, so no reported result was found on Lacuna for those comparisons.
Two ablations isolate the key ingredient. In the first, the paired replay is replaced by unpaired simulator episodes, which keeps the extra compute but loses the correlation. In the second, $\beta$ is fixed to 1, which makes the estimator a pure difference and is fragile when the simulator is poor.
Beating the field means two things. During fine-tuning we must have lower cumulative real-episode budget violations and smaller multiplier oscillation amplitude than filtered PID at matched episode count. At the end we must have a higher final return than SPiDR in regimes where SPiDR's pessimism costs reward.
Why not just…?
Why not just deploy SPiDR zero-shot and skip fine-tuning? This is the real threat. When reality lies inside the randomized ranges, SPiDR's reported 100% satisfaction may be unbeatable on safety, and we expect to tie or lose there. Our claim concerns two other regimes. The first is mismatch outside the randomization ranges, which the file lists as SPiDR's failure mode. The second is the reward lost to pessimism, since SPiDR itself notes the risk of overly cautious, "frozen" agents. We run both in-range and out-of-range mismatch grids and report honestly where zero-shot suffices.
Why not just low-pass filter the PID error signal? A filter reduces variance by averaging over past updates, which adds phase lag. The PID Lagrangian paper identifies that lag as the cause of overshoot. The control variate instead reduces variance within the current update. The filtered-PID sweep in our baselines tests this directly: if some filter time constant matches our violation and oscillation numbers, the paired estimator adds nothing.
Why not just run a safety shield during real fine-tuning? Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees uses a reachability-based backup policy and reports a 76.7% success rate and 86.7% safety rate on a physical quadruped. Shields protect individual actions, but the policy's expected-cost budget is still governed by some multiplier update. Our method improves that update and is complementary to a shield, not a replacement. We will report one shielded-plus-ours configuration to show the two compose.
Why not just tighten the budget in simulation and deploy? A fixed margin is a guess about the gap size. Too small and it is violated; too large and it wastes reward. This is the same dilemma Quanyi Li et al. describe for fixed penalties. We include a margin sweep to show that no single margin works across the mismatch grid.
Proposed Experiments
Following the modified-dynamics protocol of Reinforced Grounded Action Transformation for Sim-to-Real Transfer, the "real" system is first a held-out simulator. It has friction, mass, and actuator latency shifted both inside and beyond the training randomization ranges, which lets us run many seeds.
Tasks are drawn from Safety-Gymnasium (Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark), specifically PointGoal2 as ported to MJX by SPiDR and the velocity-constrained locomotion tasks. Safety-Gymnasium reports PPO-Lagrangian cutting violations 98% relative to PPO at a 45% reward drop on its velocity tasks. We also use Bullet-Safety-Gym SafetyCarRun-v0 and SafetyCarCircle-v0, where the adaptive primal-dual paper reports its inverse-quadratic variant best on SafetyCarRun-v0 without exact numbers.
Real-side budgets are fixed at a few episodes per dual update. We report four metrics:
- cumulative real-episode budget violations over fine-tuning;
- the number of updates until cost settles within 10% of budget;
- peak-to-peak multiplier oscillation;
- final return.
We also log the measured $\rho$ and the realized variance reduction, so the core mechanism is checked directly rather than inferred from outcomes.
If the simulated results hold, a hardware run on a 1/10-scale race car, the platform SPiDR used, provides the first physical evaluation of a PID-Lagrangian update.
The decisive checks are three. First, at matched real-episode count, our method must beat the best-tuned filtered PID on violations and oscillation in the out-of-range mismatch regime. Second, the realized variance reduction must track $1-\rho^2$, and the unpaired-replay ablation must lose the benefit. Third, as mismatch grows and $\rho$ falls, the method must degrade gracefully to plain PID rather than below it. If the first check fails, the idea is wrong. If only the third fails, the fitted-$\beta$ fallback needs redesign.
References
- Responsive Safety in Reinforcement Learning by PID Lagrangian Methods
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Adaptive Primal-Dual Method for Safe Reinforcement Learning
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark