Simulator-Informed Safe Search over Domain Randomization

Robots trained in simulation are usually made robust by domain randomization: training across many randomly perturbed copies of the simulator (different friction, masses, latencies) so the policy works on hardware it has never seen. The ranges of those perturbations are the main lever practitioners have, and they are chosen by hand. Ranges that are too narrow give policies that break on the real robot. Ranges that are too wide give policies that are timid or never learn the task. One method tunes the ranges automatically by trying policies on the real robot, but every trial is a gamble on hardware. Methods that guarantee safe trials exist, but they tune controller gains directly and ignore the simulator.

We propose to search over randomization ranges on the real robot with a safety certificate attached to every trial. The simulator acts as an informative but distrusted prior, meaning an initial belief about how safe each candidate range will be on hardware. Before deploying the policy trained under a candidate range, we predict its real-world constraint cost. That prediction starts from what the simulator ensemble says about that policy, and a learned correction is fitted to the real trials already run. A candidate is deployed only if a pessimistic bound on its real cost is below the limit. The weight given to the simulator's opinion is not decayed on a fixed schedule. It is a quantity estimated from real data, with its own uncertainty, so the certificate stays valid even when the simulator is systematically optimistic.

Core Claim

The safety of a policy trained under randomization parameters $\phi$ is a function of $\phi$ (here, means and widths of a handful of physical-parameter distributions, plus the pessimism weight described below). On the real robot that function is unknown and expensive to evaluate. In simulation it can be estimated cheaply but with bias. Our claim is that treating the simulated estimate as a learned-weight prior mean inside safe Bayesian optimization does three things. It enlarges the set of candidates that can be certified safe from the first trial. It lets the search reach higher real-world return than zero-shot pessimistic transfer within a few dozen hardware rollouts. It keeps cumulative real-world violations at or near zero, including when the simulator is wrong in the optimistic direction. Concretely, we model the real constraint cost as

$$g_{\text{real}}(\phi) = \rho \, m_{\text{sim}}(\phi) + h(\phi),$$

where $m_{\text{sim}}(\phi)$ is the policy's constraint cost under the upper quantile of a held-out wide simulator ensemble, $\rho$ is a scalar with a Gaussian prior centred at one, and $h$ is a Gaussian-process residual (a flexible random function whose uncertainty shrinks near observed points). A candidate is deployable only if

$$\mu_n(\phi) + \beta_n \sigma_n(\phi) \le d,$$

where $\mu_n$ and $\sigma_n$ are the posterior mean and standard deviation of $g_{\text{real}}$ after $n$ real trials, $\beta_n$ is the confidence multiplier from safe Bayesian optimization, and $d$ is the constraint limit. The uncertainty in $\rho$ enters $\sigma_n$ directly. As a result, the simulator loses influence only as fast as the real data contradicts it, and never faster.

How this differs from the closest work

Four existing methods define the space this proposal sits in.

Bayesian Domain Randomization for Sim-to-Real Transfer (BayRn) supplies the outer loop. It runs Bayesian optimization over the mean and variance of the randomization distribution using real rollout returns. That search is safety-blind and starts with no prior over $\phi$. We keep its bilevel structure but add a constraint model and a simulator prior.

Bayesian Optimization with Safety Constraints (SafeOpt-MC) and Meta-Learning Priors for Safe Bayesian Optimization supply the certification method. They grow a safe set using Gaussian-process confidence bounds, and the second learns kernel priors from offline real data. Both tune controller gains directly, and neither uses a simulator as a source of prior belief. We apply their safe-set logic to a different search space: randomization parameters rather than gains.

SPiDR motivates the problem. It achieves zero-shot safety by penalizing the cost with simulator-ensemble disagreement, scaled by a hand-tuned weight $\lambda$. It never uses real data, and it fails if reality lies outside the ensemble's support. We include $\lambda$ inside $\phi$, so its tuning becomes part of the certified search rather than a heuristic.

The prior-injection work, πBO and DynMeanBO, supplies the idea of trusting a prior early and recovering from a bad one later. It also constrains our design. Both decay the prior on an iteration-count schedule. In an objective-only search that is harmless. In a safety model, a schedule that keeps trusting an optimistic prior for $k$ iterations licenses $k$ unsafe deployments. Our learned weight $\rho$ is the change that makes prior injection compatible with a safety certificate.

Provably Safe Sim-to-Real Transfer is related in spirit, since it spends real samples only where simulator and reality disagree. It is tabular and uses a single nominal simulator. We work with continuous control and a randomized ensemble, and we certify at the level of policies rather than individual transitions.

Research Program

The first component is an amortized lower level. BayRn retrains a policy for every outer-loop candidate, which is the main cost that makes such a search impractical. We will instead train a single constrained policy, using PPO-Lagrangian (proximal policy optimization with a learned penalty multiplier on the constraint), conditioned on $\phi$. During training, $\phi$ is sampled per batch, and the dynamics are then sampled from the distribution $\phi$ defines. Fixing $\phi$ at deployment then yields the policy for that candidate with no retraining. We will check that the conditioned policy matches separately trained policies at a few anchor values of $\phi$. If it does not, we fall back to warm-started fine-tuning.

The second component is the simulator prior. For each $\phi$, we evaluate its policy on a held-out simulator ensemble that is wider than any training range. We record the upper quantile of constraint cost and the return, which gives $m_{\text{sim}}(\phi)$ and a matching return prior. The search space has between six and nine dimensions: friction mean and width, mass width, motor-strength width, latency width, and $\lambda$. This stays inside the low-dimensional regime where safe Gaussian-process search is known to work.

The third component is the certified outer loop. It is SafeOpt-style safe-set expansion over $\phi$, with the model above for the constraint and a residual-on-prior model for return. The initial safe seed is the SPiDR configuration with a large $\lambda$, which the SPiDR results suggest is safe on hardware. Analytically, we will state the condition under which the certificate holds: the residual $h$ has bounded complexity (bounded norm in its kernel space), and $\rho$ lies in a bounded interval. We will also show that a fixed-schedule decay violates this condition when the simulator is optimistic.

Baselines

The naive reference is uniform domain randomization over hand-chosen wide ranges, trained with the same constrained learner and the same simulation budget. The SPiDR paper reports that this reference produces 26.1× more real safety violations than SPiDR on the Unitree Go1 and 6.2× more on the RaceCar SPiDR. We expect it to be clearly beaten on violations. It is included because it can still win on return.

The strongest safety competitor is SPiDR itself. It is run with its released recipe, and its $\lambda$ is tuned in simulation with the same total simulator compute we spend on amortized training. The SPiDR paper reports 100% real-world constraint satisfaction on both the Go1 and the RaceCar. On Lacuna we found no reported real-world return value for SPiDR. We will therefore measure its return ourselves on identical hardware, terrains and episode counts. Beating SPiDR means equal or near-zero violations together with a statistically higher real return, achieved within a hardware budget we fix in advance at 30 trials.

The strongest return competitor is BayRn, given the same 30-trial budget and the same amortized lower level, so the comparison isolates the outer loop. BayRn needed 700 real rollouts across 20 policies to transfer on its own benchmarks Bayesian Domain Randomization for Sim-to-Real Transfer. That paper gives no return figure for us to compare against. Beating BayRn means matching its final return with far fewer cumulative violations.

A fourth baseline fits the randomization distribution offline, in the style of DROPO, from a short logged trajectory, and then trains SPiDR on the fitted distribution. The DROPO paper reports about 2 cm target error against about 10 cm for uniform randomization on Franka pushing.

Two ablations isolate the key ingredient. The first is safe search with no simulator prior (plain SafeOpt over $\phi$). The second is safe search with a fixed-schedule decayed prior in the DynMeanBO style. All methods share the same hardware, seeds, tuning budget and deployment protocol.

Why not just…?

Why not just deploy SPiDR and stop? It is already safe on these robots. This is the real threat. If SPiDR's return is close to the best certified-safe return, the outer loop is not worth its hardware cost. Two experiments settle it. The first compares real return at equal safety on the Go1 and the RaceCar. The second uses a condition in which the real system lies outside the ensemble's support, which is the failure mode SPiDR's own analysis leaves open. We create it by adding latency beyond the training range, and in sim-to-sim tests by holding out an entire dynamics effect. If SPiDR matches us on return in both conditions, we will report that as the finding.

Why not just fit the distribution offline with DROPO and skip online trials entirely? Offline fitting avoids risky hardware trials altogether. Its objective is the likelihood of the logged trajectories, not the task's safety or return. The BayRn paper shows that the return-driven search succeeded on ball-in-a-cup where trajectory matching converged to non-physical parameters. That evidence is indirect, so we run DROPO+SPiDR head-to-head. We also test DROPO's estimate as an extra source of prior mean, which would make the two methods complements rather than rivals.

Why not just add a constraint penalty to BayRn's objective? A penalty changes which candidate the search prefers. It does not stop the search from deploying an unsafe candidate in order to learn that it is unsafe. Cumulative violations during the search will show this directly. The tabular evidence in Provably Safe Sim-to-Real Transfer points the same way: unconstrained adaptation was unsafe in 68.5% of iterations there.

Why not just use plain SafeOpt over $\phi$ without a simulator prior? It keeps the guarantee, but it starts from a single seed and must expand the safe set slowly. Our hypothesis is that this costs most of the hardware budget, and the no-prior ablation measures exactly that.

Proposed Experiments

Development runs in sim-to-sim on Safety Gymnasium Safety Gymnasium locomotion and velocity-constrained tasks, and on simulated Go1 and RaceCar models. In each case, a held-out "real" simulator with perturbed and partly out-of-support dynamics stands in for hardware, so hundreds of seeds are affordable. Hardware evaluation follows on the Unitree Go1 and the 1/10-scale RaceCar used by SPiDR, with the Furuta pendulum from BayRn as a low-risk check of the outer loop. We measure cumulative constraint violations during the search, final real return, and the number of trials needed to reach a given fraction of the best certified return. We also measure safe-set volume over time and the calibration of predicted versus observed real cost.

Three decisive checks follow. The first deliberately biases the simulator prior to be optimistic, for example by under-reporting latency. The learned-weight model must stay violation-free while the fixed-schedule ablation incurs violations. If both stay safe, the central technical claim is wrong. The second is the no-prior ablation: it should need clearly more trials to reach SPiDR-level return. If it does not, the prior adds nothing. The third is the out-of-support condition, which is where we expect the largest return gap over SPiDR. If no gap appears there, we conclude that zero-shot pessimism suffices for these platforms.