Calibrating Sim-to-Real Safety Pessimism on Held-Out Simulators

Robots trained in simulation often break their safety constraints when they reach real hardware, because the simulator's physics is never quite right. One effective fix trains a policy against many randomized simulators and penalizes actions where those simulators disagree about what will happen. It works well, but it adds a new knob: how strongly to penalize disagreement. Set it too low and the robot is unsafe. Set it too high and the robot refuses to move. Today this knob is tuned by hand, per task. We propose to replace it with a calibration rule. Keep some randomized simulators out of training, run the current policy on them, and raise or lower the penalty until the policy's pessimistic cost estimate covers the cost it actually incurs on those unseen simulators at a chosen rate, such as 95% of the time.

The practical payoff is that practitioners would choose an interpretable coverage level instead of a raw penalty scale. We expect that level to transfer across tasks and robots where a raw scale does not. This claim can be checked directly with an existing metric for how much an algorithm suffers when one hyperparameter setting is used everywhere. The proposal sits where four concerns meet. Domain randomization (training across many sampled versions of the physics) is the setting. Safe exploration, posed as a constrained optimization problem, supplies the objective. Hyperparameter sensitivity is the failure we target and the yardstick we report. Manual reward engineering is the shortcut we refuse: the safety cost stays a separate signal and is never folded into the reward.

Core Claim

SPiDR trains on an ensemble of domain-randomized simulators and replaces the safety cost $c(s,a)$ with $c(s,a) + \lambda\,\upsilon(s,a)$, where $\upsilon$ is the disagreement among ensemble members about the next state. Its authors report 100% real-world constraint satisfaction on a Unitree Go1 quadruped and a 1/10th-scale RaceCar. They also state that $\lambda$ needs a heuristic tuning process they hope to automate, and that an overly large $\lambda$ can freeze the policy.

Our claim is that the correct $\lambda$ has an operational definition. It is the smallest value whose pessimistic cost estimate upper-bounds the cost actually incurred under dynamics the policy did not train on. To make this concrete, let $\hat C_\lambda(\pi)$ be the expected episode cost of policy $\pi$ under the training ensemble plus $\lambda$ times its accumulated disagreement. Let $C_k(\pi)$ be the episode cost measured on held-out simulator $k$, for $k = 1,\dots,K$. Let $\delta$ be the tolerated miss rate. We set

$$\lambda^\star(\pi) = \min\Big\{\lambda \ge 0 : \tfrac{1}{K}\textstyle\sum_{k=1}^{K} \mathbf{1}\big[C_k(\pi) > \hat C_\lambda(\pi)\big] \le \delta\Big\}$$

and track $\lambda^\star$ during training as the policy changes.

The constrained solver underneath, whether a Lagrangian or PID-Lagrangian method, is unchanged. It enforces $\hat C_\lambda(\pi) \le \epsilon$ for the cost budget $\epsilon$. What changes is that the free scale $\lambda$ is replaced by $\delta$, which means the same thing on every task. The mechanism borrows its spirit from Validation Pessimism Learning, which holds out a buffer of transitions to adapt a value-pessimism coefficient online in unconstrained actor-critic learning. We differ in what is held out: whole physics models rather than transitions. We also differ in what is calibrated: a safety-cost bound under dynamics shift rather than bias in value estimates.

Positioning Against Prior Work

The motivating evidence comes from three places. SPiDR shows that plain domain randomization gives 26.1 times more real violations on the Go1 and 6.2 times more on the RaceCar, so pessimism is necessary. SPiDR also leaves the scale of that pessimism unsolved. The hyperparameter sensitivity study defines sensitivity $\Phi$, the performance lost by using one fixed configuration across all tasks instead of tuning per task. It shows that stabilizing tricks can raise $\Phi$. That study evaluated only unconstrained PPO, and it never examined whether safety machinery carries the same hidden tuning burden.

Other work constrains the design. Safety Gymnasium shows that Lagrange multipliers oscillate and that PID control damps the oscillation at the cost of three extra gains. We therefore leave the multiplier controller fixed at library defaults and avoid adding a second adaptive loop with its own gains. Breaking Safety Paradox with Feasible Dual Policy Iteration warns that safe policies stop producing violation data, which degrades cost estimates. In our setting this means held-out cost estimates become noisy late in training. For that reason the calibration uses whole-episode costs on fresh held-out rollouts, which are cheap in simulation, rather than replayed transitions.

Sim-to-Lab-to-Real also calibrates safety against held-out environments, using PAC-Bayes certificates. It needs hundreds of physical "lab" rollouts and a hand-set shield threshold. We stay fully zero-shot and calibrate only on simulators. Provably Safe Sim-to-Real Transfer gives zero unsafe iterations but only in tabular problems. We give up guarantees in exchange for continuous control.

One limitation follows directly from this design. Held-out simulators drawn from the training distribution can only calibrate against variation inside that distribution. Real physics may lie outside it, the out-of-distribution failure mode noted for domain randomization. We test this explicitly by drawing a second held-out pool from a widened range.

Research Program

We will implement the calibration as a wrapper around SPiDR's ensemble-disagreement cost, with a constrained PPO learner from the SafePO library. The domain-randomization range is split once. Training simulators feed the ensemble. Held-out simulators, $K$ between 8 and 32, are used only for evaluation rollouts every fixed number of updates. After each evaluation, $\lambda$ is set by a one-dimensional search for $\lambda^\star$ on the latest held-out costs, followed by exponential smoothing to keep it from jumping. This search is cheap because $\hat C_\lambda$ is linear in $\lambda$.

The analysis has three parts. First, we track $\lambda^\star$ over training to see whether a single fixed $\lambda$ could ever have matched it, which is the case for adaptation. Second, we measure how coverage on the held-out simulators predicts coverage on a separate "pseudo-real" pool of physics models placed outside the training range. Third, we compute $\Phi$ for SPiDR with fixed $\lambda$ and for our method with fixed $\delta$, across the same task set, using the protocol of the sensitivity study at much smaller scale. That study used 4.3 million runs, which we cannot afford. We use 5 values per knob, 6 tasks, and 10 seeds.

Baselines

The naive reference point is uniform domain randomization trained with an unconstrained or Lagrangian learner and no disagreement penalty. SPiDR reports that this baseline incurs 26.1 times more real violations on the Go1 and 6.2 times more on the RaceCar. Any method that does not clearly beat it on pseudo-real violations has failed.

A second reference is manual reward shaping, with a fixed negative reward per cost event and no constraint. SafeDreamer reports an average episode cost of 14.1 for PPO-based reward shaping on MetaDrive, against 2.2 for its own method. We include shaping with the penalty swept over the same 5-value budget as our other knobs. Its purpose is to show the cost of the shortcut we decline.

The strongest competitor is SPiDR itself, with $\lambda$ tuned per task by grid search on pseudo-real-free validation, given the same ensemble size, environment steps, and learner. The only published numbers are its hardware results: 100% constraint satisfaction, plus the violation ratios above. There is no reported result on Lacuna for SPiDR's simulated RWRL or MJX PointGoal2 scores, so we will reproduce them.

Beating SPiDR means one of two outcomes. The first is that our method with one $\delta$ shared across all tasks matches per-task-tuned SPiDR on pseudo-real constraint satisfaction, within one standard error, at equal or higher return. The second is that our method shows a lower $\Phi$ than SPiDR with one shared $\lambda$.

Two ablations isolate the key ingredient. The first is static calibration: choose a single $\lambda$ once from held-out costs of a policy trained with a pilot $\lambda$, then freeze it. The second is calibration against held-out transitions rather than held-out simulators, which mirrors VPL directly. If the static version ties the online version, the contribution is the calibration criterion rather than the adaptation. If the transition-level version ties the simulator-level version, the held-out-physics idea is unnecessary.

Why not just…?

Why not just grid-search $\lambda$ against held-out simulators once, offline? This is the most serious threat, and it is the static-calibration ablation above. It needs the same held-out pool and roughly 5 times the training compute. We expect it to fall short only if $\lambda^\star$ drifts substantially during training, which our tracking analysis will reveal. If it ties, we will report the held-out coverage criterion as the finding and the offline rule as the recommended cheap implementation. That would still answer SPiDR's open tuning question.

Why not just take the worst case over ensemble members instead of mean plus scaled disagreement? Doing so removes $\lambda$, but it hides a scale in the ensemble size and the sampling range. RRLS reports that worst-case methods such as M2TD3 give up nominal performance, while domain randomization wins on average-case performance. We include a max-over-ensemble cost as an additional baseline. If it achieves equal coverage at equal return without held-out simulators, our method is redundant.

Why not just rely on a PID Lagrangian to keep cost under budget? A multiplier only enforces the budget on the cost it is shown. If that cost comes from simulators that agree with each other but are wrong about reality, a perfectly tuned multiplier still yields real violations. This is the plain domain-randomization failure SPiDR measured. The PID controller is part of every arm in our study, so this comparison is built in.

Why not just widen the randomization range? The domain-randomization literature in the file notes that overly broad ranges cause conservatism or learning failure, while overly narrow ranges miss reality. Widening the range moves the tuning problem from $\lambda$ to the range width. We sweep the width as a fixed-budget baseline and report its $\Phi$.

Proposed Experiments

The simulated testbeds are the ones SPiDR used: Safety Gymnasium PointGoal2 ported to MuJoCo JAX, and the Real-World RL suite's Cartpole, Walker, and Humanoid with their constraint channels. We add Safety MuJoCo tasks wrapped with Robust Gymnasium environment disruptors, which supplies parameter shifts on tasks with cost signals. In each task, mass, friction, damping, and actuator latency are randomized. The pseudo-real pool is drawn from a band 25–50% beyond the training range, which simulates reality falling outside the distribution.

The metrics are pseudo-real constraint satisfaction rate, episode cost relative to the budget, normalized return, the fraction of runs that freeze (return below 10% of the unconstrained policy), and $\Phi$ across tasks for each method's single knob.

The decisive checks are three. Does held-out coverage at 95% predict pseudo-real coverage within a few points, both inside and outside the training range? Does one $\delta$ shared across all six tasks beat one shared $\lambda$ on $\Phi$? Does the online rule avoid frozen policies where large fixed $\lambda$ values do not? If budget allows, a single hardware replication on a 1/10th-scale RaceCar would compare shared-$\delta$ calibration with tuned SPiDR on violation counts.