Disagreement-Shaped Priors for Safe Real-World Tuning After Sim-to-Real Transfer

When a robot controller trained or tuned in simulation reaches real hardware, two families of methods take incompatible attitudes toward uncertainty. Zero-shot safe-transfer methods treat the places where randomized simulators disagree as danger zones and permanently avoid them. Safe Bayesian optimization methods treat uncertainty as the place to collect data, cautiously testing points just inside a certified-safe region so the region can grow. The first family never improves after deployment. The second ignores the simulator entirely and starts every hardware session from one hand-verified safe setting. We propose to join them in a specific way. The spread of a domain-randomized simulator ensemble defines the prior of a Gaussian process that models the real safety constraint. A Gaussian process here is a probabilistic regression model that returns a mean prediction and an error bar at every input. Hardware tuning then proceeds by safe set expansion, using that prior, while a running test checks whether the real measurements are consistent with what the simulators believe.

The bet is that simulator disagreement is the right shape for real-world uncertainty, even when it is not the right amount. Where randomized simulators agree, the robot can be certified safe from the first trial. Where they disagree, hardware data are worth the most. When reality falls outside the randomization envelope, the consistency test should detect this from measured safety margins before a violation occurs, and the method should fall back to conservative data-only certification. If this works, it removes the initial-seed requirement of classical safe tuning in most cases. It also removes the "frozen pessimism" of zero-shot methods, and it does so without trusting the simulator blindly.

Core Claim

Let $\theta$ be a low-dimensional vector of controller parameters, such as gains or impedance settings. Let $g(\theta)\ge 0$ be the real-world safety margin from one episode, for example the minimum distance to a wall or a joint-limit slack. Each of $n$ randomized simulators $i$ yields its own predicted margin $g_i(\theta)$, with ensemble mean $\bar g(\theta)$. We place a Gaussian process prior on the real margin with mean $\bar g$ and covariance

$$k(\theta,\theta') = \frac{1}{n-1}\sum_{i=1}^{n}\big(g_i(\theta)-\bar g(\theta)\big)\big(g_i(\theta')-\bar g(\theta')\big) + k_0(\theta,\theta'),$$

where the first term is the empirical covariance of the ensemble across parameters and $k_0$ is a small stationary kernel. The stationary kernel encodes discrepancies that no simulator in the ensemble represents.

Under this prior, the error bar is wide exactly where simulators disagree. It is also correlated in the way simulators disagree, so a single real measurement shifts the estimate along the directions the ensemble says are coupled. Safe expansion uses the standard lower confidence bound $\mu_t(\theta)-\beta\sigma_t(\theta)$, where $\mu_t$ and $\sigma_t$ are the posterior mean and standard deviation after $t$ hardware trials and $\beta$ sets the confidence level. The only points ever evaluated are those whose bound is nonnegative.

The second ingredient is the consistency test. After each trial we compute the standardized residual $z_t=(y_t-\mu_{t-1}(\theta_t))/\sqrt{\sigma^2_{t-1}(\theta_t)+s^2}$, where $y_t$ is the observed margin and $s^2$ is the measurement noise. We accumulate a one-sided sequential statistic on $z_t$. If that statistic signals that reality is systematically worse than the ensemble predicts, we inflate the amplitude of $k_0$ and recompute the safe set. In the extreme case, the safe set shrinks to the region certified by real observations alone under $k_0$.

Our claim is that this prior reaches near-optimal safe parameters in substantially fewer hardware trials than both tabula-rasa safe tuning and the obvious alternative of "simulator mean plus stationary residual Gaussian process." We further claim that the consistency test keeps violations at zero across a measurable band of out-of-envelope shifts.

Relation to Prior Work

Two papers supply the method. The SafeOpt line (Safe Exploration for Optimization with Gaussian Processes; Safe controller optimization for quadrotors with Gaussian processes) supplies the expansion loop. It evaluates only points whose lower confidence bound is safe and grows the set as error bars shrink. Its hardware demonstration tuned two quadrotor gains in 30 flights with no crashes. It also exposes the two limits we target: it needs a known safe seed, and it becomes unsafe when the kernel length-scale is misspecified. Our prior replaces the hand-chosen stationary kernel with one shaped by the simulators, which is aimed directly at that misspecification failure. STAGEOPT showed that separating the expansion stage from the optimization stage improves sample efficiency, and we adopt that two-stage schedule unchanged.

Two papers motivate the work by documenting the gap from the sim-to-real side. SPiDR adds ensemble next-state variance as a penalty on the constraint cost. It achieves zero-shot constraint satisfaction on a Unitree Go1 and an RC car, but it never adapts after deployment, and its penalty weight is tuned by hand. We reuse its ensemble as a prior rather than as a fixed penalty, so disagreement becomes something to resolve on hardware instead of a permanent wall. Provably Safe Sim-to-Real Transfer shows, in tabular constrained MDPs, that real-world sample cost scales with the size of the region where the simulator is wrong. It assumes a discrete "either exact or off by at least $\sigma_s$" shift. Our continuous analogue is that hardware trials concentrate where ensemble covariance is large, with smooth rather than separable mismatch.

Several papers constrain the design. GoSafeOpt escapes disconnected safe regions using a backup-policy library and known Lipschitz constants, all learned from scratch on hardware. It is our strongest real-world competitor, and its backup mechanism is complementary rather than competing. LyEvO and Sim-to-Lab-to-Real certify safety before deployment, using statistical model checking at large compute cost and PAC-Bayes bounds over many lab environments respectively. Neither updates its safe envelope online, which is precisely what our method does. Amortized Safe Active Learning uses Gaussian processes as a simulator of functions in order to pretrain a fast query policy, giving up formal guarantees. We keep exact posterior inference, which is affordable in the episodic, low-dimensional setting we target.

Research Program

We restrict ourselves to episodic tuning of controllers with 2 to about 10 parameters. This is the regime where exact Gaussian process inference is cheap and where the cited hardware results live. Scaling to full neural-policy weights is out of scope.

The program has three parts. First, we build the ensemble-shaped prior from $n\ge 8$ randomized simulators; SPiDR reports negligible overhead at that size. We use the ensemble's empirical covariance, a low-rank matrix evaluated on a dense parameter grid, plus $k_0$.

Second, we design and analyze the consistency test. We will derive the conditional guarantee the method actually has. If the real margin is a draw from the stated prior, the usual high-probability safety argument goes through. If reality lies outside the envelope by an amount $\Delta$, the test detects the shift before a violation with a probability that depends on $\Delta$, $\beta$, and how much margin the early queries leave. This is a conditional statement rather than an unconditional guarantee, and we will report it as such.

Third, we use a query rule for the first trials that picks points with the largest ensemble-certified margin. This way, early measurements double as low-risk probes of the consistency test.

Baselines

The naive reference point is zero-shot deployment of the ensemble-mean-optimal parameters with no hardware tuning, plus plain domain randomization. On the Go1, SPiDR reports that plain domain randomization produced 26.1 times more safety violations than SPiDR, and 6.2 times more on the RC car. We include zero-shot SPiDR-style pessimistic selection as the stronger zero-shot point: it picks parameters that are safe under every ensemble member. To beat it, our method must reach strictly better task performance after tuning with zero violations.

The strongest existing online method is tabula-rasa safe tuning. We run SafeOpt with STAGEOPT staging and GoSafeOpt with its backup library, both seeded with the same verified-safe parameter we would give a human operator. GoSafeOpt reports 100% safety while tuning three parameters on a Franka Panda tracking a trajectory near a wall, and "nearly 100% safety" in MuJoCo tasks up to 11 dimensions. SafeOpt on the AR.Drone reports reaching the global optimum of a 2-parameter grid in 30 flights with zero crashes. To beat these, we must match their zero-violation record while needing materially fewer trials to come within a fixed tolerance of the best safe performance. We set that tolerance at 5% of the best safe return found by any method.

A third baseline is a sim-to-real CMDP learner that fine-tunes on hardware: PPO-Lagrangian starting from the SPiDR policy. The CMDP is a constrained Markov decision process, which maximizes return subject to a budget on expected cost. A Lagrangian method enforces the budget by adding the cost to the reward with a weight that is adjusted during training. This baseline represents the "just keep training" route.

The key ablation is the obvious idea: a simulator-mean prior with a purely stationary residual kernel whose hyperparameters are fit from the same pre-deployment data. This isolates whether ensemble-shaped covariance matters. A second ablation removes the consistency test, which isolates its contribution under out-of-envelope shifts.

All methods share the same simulators, the same hardware or "real" environment, the same trial budget, and the same evaluation episodes. Each method gets the same hyperparameter-tuning budget, carried out only in simulation against held-out ensemble members, never on the "real" system. We found no reported result on Lacuna for any of these baselines on the exact held-out-dynamics tuning tasks we construct, so all numbers there will be produced by us.

Why not just…?

Why not just use the simulator mean as the prior and model the residual with a standard Gaussian process? This is the real threat, and it is the main ablation. When the gap is a smooth, roughly uniform offset, a stationary residual kernel may perform as well as ours. Our method should win only when the discrepancy is non-stationary, meaning it is large in some regions of parameter space and negligible in others. The SafeOpt quadrotor paper's observation that a misspecified length-scale yields overconfident bounds is the mechanism by which the stationary version should fail there. We will construct both kinds of gap. If the stationary version matches ours on non-stationary gaps as well, the ensemble-shaped prior is not worth its complexity, and we will say so.

Why not just deploy SPiDR zero-shot and stop? SPiDR's guarantee comes from never entering the regions where its simulators disagree. We will measure the performance left on the table: the gap between SPiDR's chosen parameters and the best parameters that are truly safe on the "real" system. If that gap is negligible on a task, there is no case for online tuning on that task, and we will report which tasks fall into that category.

Why not just run GoSafeOpt or SafeOpt on hardware from scratch? In 2 to 3 dimensions these methods are already efficient: 30 flights on the AR.Drone. They may be hard to beat by a meaningful margin there. The case for a simulator prior rests on the higher-dimensional end, around 6 to 10 parameters. It also rests on settings where no safe seed is known, which is a hard precondition for the SafeOpt family. The tabular result in Provably Safe Sim-to-Real Transfer, where 77,958 real samples were needed with a simulator versus 296,910 without, suggests the benefit exists, but it was obtained in a very different setting. Our trial-count curves across dimensions will show where, if anywhere, the advantage appears.

Why not just fine-tune with a Lagrangian constrained RL method on hardware? These methods typically satisfy constraints only at convergence. Safety Gymnasium documents Lagrangian oscillation between aggressive and sluggish behavior unless the multiplier update is specially stabilized. We will count violations during learning for the SPiDR-initialized PPO-Lagrangian baseline, not only at the end of training.

Proposed Experiments

The simulation study uses the Real-World RL tasks from the SPiDR suite (Cartpole, Walker) and velocity-constrained MuJoCo tasks from Safety Gymnasium. The tuned object is a parameterized controller with 2, 4, 6, or 10 parameters. The "real" system is a held-out dynamics instance in one of two forms. In-envelope instances are drawn from the randomization range. Out-of-envelope instances come from the Environment-Disruptor of Robust Gymnasium. That disruptor's structural changes, including its LLM-guided adversarial shifts, give graded discrepancies the ensemble never saw. We also add an action-latency disruption, since Robust Gymnasium found that frequent low-intensity disruptions degrade policies more than rare shocks.

The metrics are the number of safety violations (target zero), the number of trials to reach within 5% of the best safe return, the final certified-safe volume, and the empirical coverage of the lower confidence bound on held-out parameters.

The decisive checks are three. First, on non-stationary in-envelope gaps, the ensemble-shaped prior should need clearly fewer trials than the stationary-residual ablation at equal violation count; otherwise the core claim fails. Second, we sweep the out-of-envelope magnitude $\Delta$ and plot violation probability with and without the consistency test. The test should hold violations at zero up to a characterized $\Delta$ and degrade gracefully beyond it, and the sweep locates exactly where the method stops being safe. Third, on tasks with no known safe seed, our method should certify a nonempty initial set in the large majority of in-envelope instances. In the same instances the SafeOpt family cannot start.

If the simulation results pass all three checks, a hardware stage will replicate the GoSafeOpt Franka wall-tracking setup, with three parameters. It will also replicate an AR.Drone-style step-response tuning, extended from two gains to four. Both use a simulator ensemble built by randomizing mass, friction, and latency. Hardware trials to near-optimal performance and the crash count will be compared directly against the figures those papers report.