Robots trained in simulation are usually made robust by domain randomization: during training, each simulated episode uses different masses, frictions, motor gains and delays, so the learned policy works across a family of plausible physics. The best recent safety methods built on this idea are pessimistic. They treat disagreement among the randomized simulators as danger and steer the robot away from it. This works when the real robot lies inside the randomized family. It has two costs. The robot avoids exactly the states where it could learn something, so it grows timid. And it has no response when the real robot behaves in a way no simulator predicted, because the simulators can agree with each other and all be wrong together.
We propose a runtime safety filter that turns the randomized simulators into a calibrated prior over the real dynamics rather than into a penalty. The filter is a small check that runs on every control step and overrides the policy's action with a fallback when that action looks unsafe. It predicts the next state using the simulators' average, together with an uncertainty band set by how much they disagree. A small online correction model is updated only when a real transition falls outside that band, which is direct evidence that reality has left the simulated family. In-family uncertainty is therefore handled by the filter rather than by making the policy timid. Out-of-family surprises are detected and learned from, using a handful of hardware samples concentrated where the simulators are wrong.
Core Claim
The ensemble of randomized simulators already contains a state-dependent map of where physics is uncertain. That map is currently used only to discourage visits. It is more useful as the prior mean and prior variance of a residual model inside a predictive safety filter, with an event trigger that collects real data only on out-of-envelope transitions. This is a continuous-state version of the "learn only in the mismatch region" principle that has so far been proven only for tabular problems. We expect it to keep the zero-violation behavior of pessimistic randomization when reality is in-family. We also expect higher task performance in-family, safe recovery when reality is out-of-family, and a real-sample cost that grows with the size of the mismatch rather than with the size of the state space.
Where This Sits
Two papers motivate the gap from opposite sides. SPiDR trains with at least eight randomized simulators and adds their next-state variance to the safety cost, $\tilde c = c + \lambda \upsilon(s,a)$. It reports 100% real-world constraint satisfaction on a Unitree Go1 and a 1/10-scale race car. Its authors acknowledge three limits: large $\lambda$ makes the robot freeze, $\lambda$ is set by hand, and nothing adapts if reality falls outside the randomized support. Provably Safe Sim-to-Real Transfer shows the opposite principle. Real data must be collected inside the mismatch region $B$, and with a safe baseline policy this can be done with zero unsafe iterations and sample cost scaling with $|B|$. That result holds only for finite state and action spaces, under an assumption that the simulator is either exact or wrong by at least a fixed margin, and with a single simulator rather than a randomized family. Our proposal is the continuous counterpart. The randomized family shrinks the effective mismatch region, and the out-of-envelope test replaces the separability assumption.
Three papers supply the method. Feedback Linearization based on Gaussian Processes with event-triggered Online Learning updates a Gaussian process only when uncertainty crosses an error-dependent threshold. A Gaussian process is a nonparametric regressor that returns both a prediction and a calibrated error bar. We reuse this trigger to keep the model small, given the cubic cost of exact updates. Physics-Informed Kernel Embeddings puts a physical model into the prior of a kernel regressor to cut sample needs. We replace its single physical model with the randomized ensemble's mean and spread. Safe Learning-Based Feedback Linearization Tracking Control (CLBFET) filters actions through a barrier-function program driven by a Gaussian-process disturbance estimate. It starts from a nominal model, uses no simulator family, and was tested only in simulation. Our filter is closest to it, and our central ablation isolates what the randomized prior adds over CLBFET's nominal-model prior.
Two papers constrain the design. The feedback-linearization literature, notably Feedback Linearization for Uncertain Systems via Reinforcement Learning, notes that its guarantees degrade when the nominal model is badly wrong. Exact cancellation of nonlinearities is also at odds with physics that varies across environments. So we do not linearize. The filter reasons directly over predicted state tubes. Gameplay Filters shows that rollout-based filters with a fallback policy work zero-shot on full-order quadrupeds. It obtains robustness from a learned adversary that has no convergence guarantee and never updates after deployment. We keep its rollout-plus-fallback structure and replace the adversary with a data-calibrated uncertainty tube that does update.
Research Program
The prior is built in simulation. We train $n$ randomized simulators, as in SPiDR, or use the randomized simulator itself to obtain, for each state $s$ and action $a$, an ensemble mean $\mu(s,a)$ and covariance $\Sigma(s,a)$ of the next state. Randomized latency is part of the family, following the 15–19 ms delay modeling of Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. The real next state is modeled as
$$s' = \mu(s,a) + r(s,a) + w, \qquad r \sim \mathcal{GP}\big(0,\; k_0 + \alpha\, \mathrm{tr}\,\Sigma(s,a)\, k_1\big),$$
where $r$ is the unknown residual, $w$ is sensor noise, and $k_0, k_1$ are kernels. The term $k_0$ is a small residual allowance everywhere. The term $\mathrm{tr}\,\Sigma \cdot k_1$ lets the residual be large only where the simulators disagree, and $\alpha$ sets the trade-off between the two. In-family uncertainty is thus encoded before any real data arrives.
The data-collection rule is the core of the method. After each real transition we compute the standardized surprise
$$z = (s' - \hat s)^\top (\Sigma + \Sigma_{\mathrm{GP}} + \Sigma_w)^{-1} (s' - \hat s),$$
where $\hat s$ is the current prediction and $\Sigma_{\mathrm{GP}}$ is the residual model's own variance. The point is added to a sparse Gaussian-process budget only if $z$ exceeds a chi-squared quantile. A large $z$ means the real transition is implausible under the whole randomized family plus current corrections. These triggered points form our continuous estimate of the mismatch region, and their count is the real-sample cost we measure.
The filter runs on every control step. It rolls the proposed action forward over a horizon $H$ that covers the actuation delay, then rolls a fixed backup policy (braking for the car, a stand-and-stabilize controller for the quadruped) for a further $H_b$ steps. It propagates a confidence tube of width $\beta$ times the combined standard deviation. The action is accepted if the whole tube stays inside the safe set, and otherwise the backup action is executed. The task policy is trained in simulation with an ordinary constrained reinforcement-learning solver from SafePO, without SPiDR's variance penalty. It may therefore approach in-family uncertain regions, and the filter alone decides whether that is safe. The confidence scale $\beta$ replaces SPiDR's hand-tuned $\lambda$, and we calibrate it on held-out randomized simulators by coverage of true next states. Calibrating $\beta$ is cheaper than tuning $\lambda$ because the target is a measurable coverage level, not a reward-safety trade-off.
The analysis asks for two guarantees. Under standard Gaussian-process bounds, if the residual lies in the kernel's function space, the tube is valid with high probability and the backup keeps the system safe. We will state this bound, and its dependence on an unknown function-space norm, honestly as a caveat. We will also bound the number of triggered points by the size of the region where the real residual exceeds the in-family envelope, mirroring the $|B|$ scaling of the tabular result.
Baselines
The naive reference is standard domain randomization with an unconstrained or reward-penalized policy and no filter. SPiDR reports that this baseline produced 26.1 times more real violations than SPiDR on the Go1 and 6.2 times more on the race car. It sets the floor and checks that our hardware setup reproduces the known failure.
The strongest existing method is SPiDR itself. On the race car it reports 100% zero-shot constraint satisfaction, and we treat that as the in-family safety bar. Gameplay Filters is the strongest competing filter, and we will run it on the same platforms with its published settings. All methods share the same randomized simulator, the same SafePO solver for the task policy, equal simulation steps, and the same hyperparameter search budget. That budget covers SPiDR's $\lambda$, the adversary strength for Gameplay Filters, and $\alpha$ and $\beta$ for ours. Beating SPiDR in-family means matching its zero violations while achieving higher task return, such as lap speed or goal reach rate, and less freezing. Beating it out-of-family means keeping violations near zero under held-out physics where SPiDR's violations rise. If SPiDR also stays safe out-of-family, the safety half of our claim fails and only the performance half remains to be tested.
On the tabular gridworld benchmark, the reference is Provably Safe Sim-to-Real Transfer. It reports 0% unsafe iterations and 77,958 real samples, against 296,910 for safe learning without a simulator. Our method, run on a gridworld with a randomized family of wind strengths, should match zero unsafe iterations with no more samples. This is a sanity check that the continuous design does not lose the tabular result.
Three ablations isolate the key ingredient. The first replaces the ensemble prior with a single nominal simulator and an isotropic residual variance, keeping the filter and trigger. This is essentially CLBFET with a rollout filter. The second keeps the ensemble prior but updates the residual model on every transition instead of on triggered surprises. The third keeps the ensemble prior and freezes it, so there is no online learning at all. The first ablation tests whether randomization helps the prior. The second tests whether selective collection matters for cost and compute. The third tests whether adaptation matters out-of-family.
Why not just…?
Why not just widen the randomization range so reality falls inside it? This is the cheapest fix and partly works, but SPiDR's own authors report that wide ranges and large penalties make the robot timid. The domain-randomization tuning study How to pick the domain randomization parameters also points to a steep simulation cost. We will sweep the range width for SPiDR and for our method. The test is whether a widened SPiDR can reach our out-of-family safety without falling below our in-family return. If it can, the online component is unnecessary and we will say so.
Why not just fine-tune the policy on real data after zero-shot deployment? Fine-tuning collects data wherever the policy goes, and constrained solvers are only safe at convergence. Breaking Safety Paradox documents violations during training. In the tabular setting, unconstrained sim-to-real learning was unsafe in 68.5% of iterations. We include SPiDR followed by constrained fine-tuning as a baseline and count violations during adaptation, not only after it.
Why not just use an online Gaussian-process safety filter from a nominal model, as CLBFET does? This is the real threat, because the filter and trigger might do all the work. The single-nominal-simulator ablation settles it directly. Our claim requires that the randomized prior reduce triggered real samples, or violations in the first minutes of deployment, at equal filter design. If the two curves coincide, the contribution reduces to a better-engineered CLBFET.
Why not just use robust worst-case control? Covering all discrepancies with a fixed disturbance bound is known to be conservative. Regularized and Distributionally Robust Data-Enabled Predictive Control shows performance collapsing as the robustness radius grows, and such bounds cannot shrink with data. The frozen-prior ablation with a fixed, coverage-calibrated tube is our version of this baseline.
Proposed Experiments
In simulation we use Safety-Gymnasium PointGoal2 ported to MJX, as in SPiDR, and the Real-World RL Suite cartpole and quadruped tasks with their delay and noise perturbations. The latter matter because An empirical investigation of the challenges of real-world reinforcement learning found combined delay and noise to break standard agents. Each task has two test splits. The in-family split draws physics from the training range. The out-of-family split holds out effects outside it: friction and mass beyond the range, an unrandomized extra actuation delay, and a localized force field analogous to the gridworld's windy cells. Metrics are episode cost and violation rate, task return, the fraction of steps the backup overrides the policy (our measure of freezing), the number of triggered real samples, and filter latency per step.
On hardware we use the 1/10-scale race car, the platform on which SPiDR reported 100% safety. Out-of-family conditions are created by changing tires to a surface outside the randomized friction range and by adding payload mass. A Go1 replication is a stretch goal, with the filter operating on a reduced state of body height, pitch and velocity.
The decisive checks are three. In-family, our method must match SPiDR's zero violations with higher return and fewer overrides. Out-of-family, our violations must stay near zero while SPiDR's and the frozen-prior ablation's rise. Triggered real samples must concentrate in the held-out mismatch region and stay well below the every-step ablation and the nominal-prior ablation. Failure on the first check means filtering costs more than penalizing. Failure on the third means the randomized prior is not what makes adaptation cheap.
References
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Feedback Linearization Based on Gaussian Processes With Event-Triggered Online Learning
- Physics-Informed Kernel Embeddings: Integrating Prior System Knowledge with Data-Driven Control
- Safe Learning-Based Feedback Linearization Tracking Control for Nonlinear System with Event-Triggered Model Update
- Feedback Linearization for Uncertain Systems via Reinforcement Learning
- Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Regularized and Distributionally Robust Data-Enabled Predictive Control
- An empirical investigation of the challenges of real-world reinforcement learning