A robot trained in simulation can be made safe on day one by being pessimistic: wherever a family of randomized simulators disagrees about what happens next, treat that state as dangerous and stay away. SPiDR shows this works on hardware. It adds the variance across randomized physics engines to the safety cost, and it reports 100% constraint satisfaction on a Unitree Go1 and a small race car. On those platforms, plain domain randomization caused 26.1x and 6.2x more violations. The price is that the pessimism never goes away. The penalty weight is fixed, nothing is learned from real deployment, and the robot stays as cautious on its thousandth real episode as on its first. If reality falls outside the randomized range, the method also has no way to notice.
This note proposes using a small number of safely collected real transitions for two jobs. The first is to shrink the safety margin only where reality has shown which simulators are right. The second is to grow the margin automatically when reality turns out to lie outside what any simulator predicted. The policy keeps training cheaply in simulation, and real data is spent only on recalibrating the margin. The bet is that this recovers most of the performance lost to zero-shot pessimism within a few dozen real episodes, without the real-world violations that grounding methods such as RGAT incur by collecting calibration data with no safety constraint.
Core Claim
Disagreement-based pessimism has two separate pieces: which simulators are still plausible, and how far reality may stray from them. Each piece should be estimated from real data with its own mechanism. Let $\xi_1,\dots,\xi_M$ be sampled simulator parameters such as friction, mass, and latency. After real transitions $\mathcal{D}_t$ have been collected, give each simulator a weight $w_m \propto \exp(\beta \log p_{\xi_m}(\mathcal{D}_t))$. Here $p_{\xi_m}$ is the likelihood of the observed real next states under simulator $m$ with a fixed noise model, and $\beta \le 1$ tempers overconfident updates. Let $\upsilon_w(s,a)$ be the weighted variance of the predicted next states. The safety cost used for continued simulated training is
$$\tilde c_t(s,a) = c(s,a) + \kappa_t\,\upsilon_w(s,a), \qquad \kappa_t = \text{the } (1-\alpha)\text{ quantile of } \frac{\lVert s'_{\text{real}} - \bar s'_w(s,a)\rVert^2}{\upsilon_w(s,a)} \text{ over } \mathcal{D}_t.$$
In this cost, $c$ is the task's safety cost, $\bar s'_w$ is the weighted ensemble mean prediction, and $\alpha$ is a target miss rate. The reweighting lets the margin relax where it is safe to do so. Parameters that real data has pinned down stop producing disagreement. Parameters that only matter in states the robot has never visited keep their full spread, so pessimism persists there. The quantile $\kappa_t$ replaces SPiDR's hand-tuned $\lambda$ with a number calibrated on real transitions. If real next states land outside the ensemble's spread, as happens when reality is out of support, $\kappa_t$ rises and the margin tightens. The method then takes on the job of SPiDR's support assumption and replaces it with a check against data.
What would be new is concrete. Zero-shot pessimism methods do not adapt online. Provably safe transfer through mismatch regions has so far been confined to tabular problems. Online tightening methods calibrate a single scalar and are not tied to simulator ensembles. This proposal adapts the tightening state by state, for continuous control, driven by the mismatch between real and simulated predictions.
Positioning
Three groups of work motivate the proposal and constrain its design. The first group is zero-shot sim-to-real safety. SPiDR penalizes ensemble disagreement with a fixed weight, and Gameplay Filters trains a virtual adversary in simulation and filters actions through short imagined games against it. Both are safe at deployment, and neither ever relaxes, because neither uses real data. We keep SPiDR's cost structure and add the online calibration loop it lacks.
The second group is tightening that shrinks as data arrives. The predictive safety filter tightens constraints using a Bayesian model's uncertainty. Learning Safety Filters for linear systems shrinks polytopic constraint sets as least-squares confidence ellipsoids contract. DOPE and Safe-PSRL (DOPE, Safe-PSRL) use shrinking pessimistic buffers in tabular constrained problems. These works supply the principle that margins should track epistemic uncertainty. They also impose a constraint on our design: all of them assume the true system lies inside the model class, and their guarantees fail when it does not. The quantile term exists to handle exactly that failure.
Online Constraint Tightening in Stochastic MPC adjusts one tightening scalar until the observed satisfaction rate matches a target. That is the closest precedent for the $\kappa_t$ loop. We calibrate on prediction residuals rather than on violation events. Violations are rare, so a loop driven by them needs long stationarity waits. Residuals arrive with every transition.
The third group, Provably Safe Sim-to-Real Transfer, is the closest in spirit. It confines real interaction to the region where the simulator and reality mismatch, and it proves that sample cost scales with the size of that region. Its experiments use a 5x5 gridworld, it requires a separability assumption, and it uses fixed linear-programming margins. Our weighted ensemble is the continuous, smooth-mismatch counterpart of its certified region. We trade its proofs for scale, and we say so.
Research Program
The system has an outer loop alternating between simulation and reality. It starts from SPiDR. A constrained policy is trained with SAC-Lagrangian on GPU-parallel MuJoCo (MJX) across $M$ randomized simulators, with uniform weights and a conservative initial $\kappa_0$. In each real phase, the current policy collects a few episodes. Because the policy was trained against $\tilde c_t$, real collection is itself tightened exploration. As an optional add-on, the action can be nudged toward states with high $\upsilon_w$ that the penalized safety critic still rates as feasible, which directs real samples toward unresolved mismatch. After each real phase, we update the weights $w_m$ and the quantile $\kappa_t$ and continue simulated training of the same policy and critics under the new cost. No real transition is used for policy gradients. This is what keeps the real sample budget small.
Three analyses accompany the system. The first measures how $\upsilon_w$ contracts over visited versus unvisited state regions. This tests whether relaxation really stays local. The second measures how $\kappa_t$ responds to injected out-of-support effects, namely actuator delay and parameters outside the randomization range. We expect it to rise within a handful of episodes; if it does not, the out-of-support claim fails. The third measures the realized real-world miss rate against the target $\alpha$. The data are not exchangeable, because the policy changes between phases, so we report empirical coverage rather than claim a formal guarantee.
Baselines
The naive reference point is plain domain randomization with a Lagrangian constrained learner and no disagreement penalty, deployed zero-shot. It shows the cost of ignoring the gap. On hardware, SPiDR reports this baseline at 26.1x (Go1) and 6.2x (race car) more violations than SPiDR.
The strongest existing method is SPiDR itself with its fixed $\lambda$, deployed zero-shot and also given the same number of real episodes as ours, used only to tune $\lambda$ by grid search. That gives it an equal real-data budget. SPiDR reports 100% constraint satisfaction on both hardware platforms. For its simulated Safety Gym PointGoal2 and RWRL returns, the file contains no reported result on Lacuna, so we rerun it with the released recipe. Beating SPiDR means two things together. Our violations, counted during adaptation as well as at the end, must be no higher than SPiDR's. Our return after the real budget must be higher, closing at least half of the return gap between zero-shot SPiDR and an oracle trained with the true parameters.
A second strong competitor is system identification followed by narrowed randomization. Here we fit the simulator parameters by maximum likelihood on the same real data, resample a tight ensemble around the estimate, and retrain with SPiDR. This is the alternative most likely to match us, as discussed below.
A third competitor is global scalar tuning in the style of the online stochastic MPC method: uniform weights, with $\kappa_t$ adjusted from observed violation rates.
The unconstrained-calibration reference is RGAT, run with the same real budget. It shows the violation cost of grounding with no constraint during data collection. The Provably Safe Sim-to-Real paper reports that its unconstrained baseline was unsafe in 68.5% of iterations, in its gridworld setting.
Two ablations isolate the key ingredients. In the first, reweighting is kept and $\kappa$ is frozen at $\kappa_0$. In the second, $\kappa_t$ is calibrated with uniform weights.
All methods share the same simulator ensemble, the same SAC-Lagrangian implementation, the same simulated step budget, the same real-episode budget, and the same hyperparameter search budget of 20 configurations, with tuning done on a held-out parameter setting.
Why not just…?
Why not just tune SPiDR's single $\lambda$ on real rollouts? A single scalar cannot relax in some places while staying cautious in others. Lowering $\lambda$ enough to recover agility on surfaces the robot has already visited also removes protection on surfaces it has not. The uniform-weights ablation is this alternative in disguise. Our prediction is that it either violates constraints on held-out terrain or stays as conservative as SPiDR. If it matches the full method, the per-simulator reweighting is unnecessary, and we would report that result.
Why not just identify the parameters and retrain with narrow randomization? This is the real threat. When reality lies inside the simulator family and the data identify every parameter that matters, it may perform just as well. It is expected to fail in two cases that we construct deliberately. The first is partial identifiability: parameters that are only excited in unvisited regions get collapsed to arbitrary point estimates. The second is out-of-support reality: a point estimate inside a misspecified family looks confident and is wrong. The experiments separate these regimes. If system identification ties us in-support and also in the partially identifiable case, the contribution shrinks to out-of-support detection alone.
Why not just learn a residual dynamics model or ground the simulator, as RGAT does? Residual and grounding models need real data from the risky states they are meant to correct, and RGAT collects that data with no constraint. Every ingredient they need can be added on top of our safe collection. We test this by running RGAT on the real transitions we collect safely, and we report whether it adds return beyond our method.
Why not just add a large penalty to the reward? SafeDreamer reports that reward-shaped PPO in MetaDrive reached an average episode cost of 14.1, against 2.2 for an explicit constrained formulation. We therefore keep constrained optimization as the base learner and do not revisit this alternative.
Proposed Experiments
The surrogate-real experiments use Safety Gym PointGoal2 on MJX and the RWRL Cartpole and Walker tasks, the settings SPiDR uses. In each, a "real" environment is one held-out simulator, so violations can be counted exactly. Three regimes are tested. The in-support regime draws the real parameters from the randomization range. The partially identifiable regime uses friction that differs only in a floor region the initial policy rarely visits. The out-of-support regime uses mass 1.4x beyond the range, echoing the mismatch in the barrier-certificate quadrotor study, plus an unmodeled actuator delay.
The metrics are cumulative real constraint violations over the whole adaptation period, the number of real episodes needed to reach 80% of oracle return, final return, and realized miss rate against $\alpha$.
Three outcomes would be decisive. First, in the partially identifiable regime, our method should match system identification's return on visited regions while keeping zero violations on the unvisited floor, where system identification violates. Second, in the out-of-support regime, $\kappa_t$ should rise and violations should stay near zero, while SPiDR and system identification both violate. Third, real-episode counts to reach 80% of oracle should fall well below those of zero-shot-then-$\lambda$-tuned SPiDR.
If the simulated results hold, a hardware check on a 1/10-scale race car, replicating SPiDR's obstacle task, will test whether the relaxation is real: faster laps with no added collisions over 30 real episodes.
References
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- Gameplay Filters: Robust Zero-Shot Safety through Adversarial Imagination
- A predictive safety filter for learning-based control of constrained nonlinear dynamical systems
- Learning Safety Filters for Unknown Discrete-Time Linear Systems
- DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
- A Safe Bayesian Learning Algorithm for Constrained MDPs with Bounded Constraint Violation
- Online Constraint Tightening in Stochastic Model Predictive Control: A Regression Approach
- Provably Safe Sim-to-Real Transfer
- Safe Learning of Quadrotor Dynamics Using Barrier Certificates