Hard Safety Filters Calibrated Over Randomized Physics

Robots trained in simulation are usually made robust by domain randomization: during training, the simulator's physics (masses, friction, motor strength, latency) are resampled every episode so that the real robot looks like one more random draw. This makes policies work on average, but it does not keep them inside hard limits. A quadruped that tips over once in fifty deployments has an acceptable average and a broken gearbox. The usual fixes add a penalty or an expected-cost budget for unsafe behavior. These are "soft" constraints, and they let the learner trade a rare crash against a large reward. We propose to replace the penalty with a runtime safety filter. The filter is a learned estimate of how close the robot is to unavoidable failure, written as a function of both the robot's state and the simulator's physics parameters. Before each command reaches the motors, the filter checks it against the worst of many sampled physics settings and overrides it if any of them predicts failure. The size of the filter's safety buffer is then set statistically, using held-out randomized simulators, so that it comes with an explicit violation-rate guarantee.

The two-sentence version is this. Domain randomization already assumes the real world is just another draw from the simulator's physics distribution. If we take that assumption seriously, it gives us exchangeable "calibration robots" for free, and we can certify a hard override filter against them without any real-world data or adversarial training. The novelty is in making the transfer assumption explicit and testable, and in using it to calibrate a hard filter instead of tuning a soft penalty.

Core Claim

Our central claim has three parts. The first is that a reachability critic conditioned on physics parameters can be trained without an adversary, because the simulator supplies those parameters as privileged information during training. A reachability critic is a value function that predicts the worst future safety margin along a trajectory, rather than a sum of discounted costs. The second is that taking the minimum of this critic over a sampled set of physics parameters gives a cheap, rollout-free worst case over physical uncertainty. The third is that calibrating the override margin at the level of whole simulated domains yields a finite-sample bound on the per-episode violation probability. That bound holds for any new domain drawn from the randomization distribution, and therefore for the real robot exactly to the extent that domain randomization's own premise holds. We expect this filter to match SPiDR's zero-shot safety at equal or higher task return. We also expect it to expose, rather than hide, the cases where the real world falls outside the randomized envelope.

Positioning Against Prior Work

The motivating evidence comes from the soft-constraint literature. Ghosh et al. (Towards Achieving Sub-linear Regret and Hard Constraint Violation in Model-free RL) show that expected-cost constraints let safe episodes "pay for" catastrophic ones. Bou-Ammar et al. (Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret) show that a scalar penalty is violated whenever the reward is large enough. Both results argue against any design whose safety rests on a tuned penalty weight.

The closest existing work is SPiDR. SPiDR adds to the safety cost the disagreement between next-state predictions across a randomized simulator ensemble, $\tilde{c}=c+\lambda\upsilon$, and then trains a constrained RL policy on that cost. It reports 100% real-world constraint satisfaction and 26.1x fewer violations than plain randomization on a Unitree Go1. We keep its central insight that the randomized ensemble is a source of pessimism, but we use the ensemble differently in three ways:

  • We use it at runtime, as an override, rather than inside a cost objective.
  • We look at worst future safety margin rather than one-step disagreement.
  • We replace the hand-tuned $\lambda$ with a margin chosen by calibration.

Gameplay Filters supplies our filter architecture: a task policy, a safety fallback policy, and an override when predicted safety is lost. It also constrains our design. Its authors note that the learned adversary has no convergence guarantee, and that an under-trained adversary certifies unsafe actions. Their filter also runs multi-step imagined rollouts at control time. We remove the adversary for parametric uncertainty by enumerating sampled physics directly. We remove the rollouts by querying a conditioned critic in one batched forward pass. We concede that the adversary covers disturbances such as external pushes, which our method does not model unless they are randomized.

Sim-to-Lab-to-Real supplies two things: the Hamilton–Jacobi safety Bellman equation for the backup shield, and the idea of a statistical certificate. However, that certificate is computed on hundreds of physical "Lab" environments. We compute it on held-out simulated domains. Provably Safe Sim-to-Real Transfer achieves zero unsafe iterations, but only in tabular settings and only with a known safe baseline policy. Our approach is continuous and learns its own fallback.

The calibration step reuses split-conformal ideas from CP-NCBF and the conformal safety buffer of Safe Flow Q-Learning. Both of those calibrate within a single fixed simulator. Conformal Policy Control warns that such guarantees fail under distribution shift. We take that warning head on: our exchangeable unit is a whole simulated domain, so the guarantee is explicitly conditional on the real robot being a draw from the randomization distribution. Out-of-support physics is a stress test we run, not a case we claim to cover.

Finally, the idea of feeding the critic privileged physics comes from Asymmetric Actor Critic. There, a critic that sees ground truth trains a deployable actor that does not, with no safety component. Here, the privileged quantity is the physics vector, and the critic remains usable at deployment because we minimize over that vector instead of observing it.

Research Program

Physics-conditioned safety critic. Let $g(s)$ be the signed distance from state $s$ to the failure set; it is negative inside failure. Let $\xi$ be the vector of randomized physics parameters and $\gamma$ a discount factor near one. We train a safety Q-function on the discounted safety Bellman equation used in Sim-to-Lab-to-Real, conditioned on $\xi$:

$$Q(s,a,\xi) = (1-\gamma)\,g(s) + \gamma \min\Big(g(s),\ \max_{a'} Q(s',a',\xi)\Big), \quad s' \sim P_\xi(\cdot \mid s,a).$$

Because $\xi$ is known in simulation, each domain is a single-player problem. A fallback policy $\pi_b(s)$ is trained to maximize the robust value $\min_{k} Q(s,a,\xi_k)$ over a fixed bank of $K$ physics samples $\xi_1,\dots,\xi_K$ drawn from the randomization distribution $\Xi$. The task policy is trained separately for reward, and without the penalty, under ordinary domain randomization.

Runtime filter. At each step, the task action $a_t$ is executed only if

$$\min_{k\le K} Q(s_t,a_t,\xi_k) \ \ge\ \delta,$$

and otherwise $\pi_b(s_t)$ is executed instead. This check costs one batched network evaluation of size $K$ and involves no rollout.

Domain-level calibration. We draw $N$ fresh domains from $\Xi$ that are never used for training. In each one, we run full shielded episodes and record whether a violation occurs at each candidate margin $\delta$ on a grid. Violation rate is only approximately monotone in $\delta$, so we select $\delta$ with a conformal risk-control or learn-then-test procedure that does not need monotonicity, in the spirit of generalized conformal risk control. The target is a violation probability of at most $\epsilon$ per episode, with confidence $1-\alpha$, for a new exchangeable domain.

This calibration is end-to-end. It absorbs every source of error in a single number: critic error, gaps between the $K$ sampled domains, and fallback imperfections. It does not certify the critic in isolation.

Analysis. We will characterize two quantities as $K$ and $N$ vary:

  • Coverage: how far the realized violation rate on held-out in-support domains sits from $\epsilon$.
  • Conservatism: the fraction of steps on which the filter overrides the task policy, and the return lost as a result.

We also measure how coverage degrades as test physics move outside the support of $\Xi$. This last curve is the honest answer to the support-mismatch problem that SPiDR assumes away.

Baselines

The naive reference point is plain domain randomization with a fixed penalty for safety violations, trained on the same randomization distribution and the same simulation budget. SPiDR reports that on hardware this baseline incurs 26.1x (Go1) and 6.2x (RaceCar) more violations than SPiDR. Any method that does not beat it by a wide margin is uninteresting.

The second baseline is PPO-Lagrangian trained under domain randomization, the standard expected-cost approach. Safety Gymnasium reports that PPO-Lag achieves a 98% reduction in violations over PPO on velocity tasks with a 45% drop in reward. We read this as the benchmark for "safety bought with return." We will beat it if we reach equal or lower violation rate with smaller reward loss.

The strongest competitor is SPiDR itself. It targets exactly this question, zero-shot safety under randomized physics, and it has real-world evidence. We will run it on the same randomization distribution and the same number of simulator steps. Its $\lambda$ will be swept over a grid of the same size as our candidate grid for $\delta$, so the two methods have equal tuning budgets. For SPiDR's simulated PointGoal2 and RWRL figures, we found no reported result on Lacuna. We will therefore reproduce SPiDR in MJX rather than quote numbers.

Gameplay Filters is the strongest runtime-filter competitor. We will train its adversary with the same number of simulation steps as our critic plus fallback, and give it the same disturbance budget. It reports 10/10 safe runs on irregular terrain and survival under 150 N tugs on hardware. We do not expect to beat it on unrandomized push disturbances. We do expect to match it on parametric uncertainty while using lower control-time compute and showing lower seed-to-seed variance.

Beating SPiDR means a lower held-out violation rate at matched return, or higher return at matched violation rate, across seeds. Beating Gameplay Filters means equal safety on parametric shifts at lower control-time compute and with lower seed-to-seed variance.

Three ablations isolate our ingredients. The first is the same filter with a critic trained on the marginal randomized dynamics, without conditioning on $\xi$, which is the obvious "reachability critic plus domain randomization" design. It tests whether conditioning and the minimum over domains matter, or whether averaging suffices. The second replaces the calibrated $\delta$ with a hand-tuned margin, testing calibration. The third replaces the minimum over domains with a mean over domains, testing pessimism.

Why not just…?

Why not just tune SPiDR's $\lambda$ higher until violations vanish? This is a real threat, since SPiDR already reports full real-world satisfaction on its two platforms. Our reply is not that SPiDR fails. It is that the penalty sits inside an expected-cost objective, so it inherits the reward-for-risk trade-off identified by Bou-Ammar et al., and its pessimism is one-step disagreement rather than worst future margin. The matched-tuning-budget sweep will settle the question. If SPiDR's best $\lambda$ reaches our violation rate at equal return on every benchmark, the remaining contribution of our work is only the explicit calibrated guarantee, and we will report it as such.

Why not just use a Lagrangian method under domain randomization? Expected-cost constraints allow safety surpluses to offset crashes (Ghosh et al.; Stradi et al.). In offline benchmarks, Lagrangian baselines such as CPQ and COptiDICE failed to satisfy constraints where the reachability method FISOR succeeded on all 26 DSRL tasks. The PPO-Lag baseline will show whether this carries over to randomized online training.

Why not just train an adversarial gameplay filter? It handles disturbances we do not model, and that is its genuine advantage. Its cost is a learned adversary without convergence guarantees, together with imagined rollouts at control time. For uncertainty that domain randomization already parametrizes, enumerating sampled physics is an exact inner minimum over the sampled set, so the adversary is unnecessary there. The comparison on seed variance and per-step latency will show whether that simplification pays off.

Why not just identify the robot's physics precisely and use a nominal barrier filter? Even the carefully identified Minitaur model of Tan et al. still needed randomization to avoid falls on hardware. Closed-form invariant-set filters such as the gauge map of Tabas and Zhang need a single linear model with known disturbance bounds, which legged robots do not provide.

Proposed Experiments

Benchmarks. We will use three settings:

  • Safety Gym PointGoal2 ported to MJX and the RWRL suite, matching SPiDR's simulation settings.
  • RRLS MuJoCo tasks (RRLS, code at SuReLI/RRLS). We use their 1D–3D uncertainty sets over mass, friction and damping, and define the failure sets ourselves: torso height and angle limits for Hopper, Walker and Ant. RRLS reports no per-task numbers on Lacuna. It does find that domain randomization is the strongest average-case baseline and that worst-case methods such as M2TD3 sacrifice nominal return, which frames our conservatism measurement.
  • The Pybullet 3D car, 6D drone and 8D quadruped tasks of CP-NCBF, wrapped with randomized physics, to test scaling with dimension.

Metrics. For every method, we report per-episode violation probability on 1,000 held-out in-support domains with confidence intervals, normalized return, override rate, and per-step filter latency. All results are over ten seeds.

Decisive checks. There are four:

  1. Calibration should hold. The realized violation rate on held-out in-support domains should stay at or below $\epsilon$ (we will test $\epsilon=0.01$ and $0.05$) as $N$ varies, and it should be visibly violated when calibration is skipped.
  2. At matched return, the conditioned worst-case filter should show fewer violations than both SPiDR and the unconditioned critic ablation. If the unconditioned ablation matches it, conditioning on physics is not the active ingredient, and we will say so.
  3. Under out-of-support shifts, where friction and mass are scaled progressively beyond the bounds of $\Xi$, we expect coverage to break down. Our filter's override rate should rise before violations rise, providing a usable warning signal. We will check whether it does. If violations rise without overrides, the method offers no protection beyond the randomized support.
  4. If a 1/10-scale car or small quadruped is available, we will run a hardware zero-shot test against SPiDR with a pre-registered $\epsilon$. This mirrors SPiDR's RaceCar evaluation, where plain randomization incurred 6.2x more violations.