Belief-Weighted Pessimism for Safe Zero-Shot Sim-to-Real Transfer

Robots trained in simulation often break things when first moved onto real hardware, because the simulator's friction, mass, and delays are wrong. A recent fix is to train in many randomized simulators, measure where those simulators disagree about what happens next, and charge the robot an extra safety cost for entering those regions. This keeps the robot safe on day one. The cost, however, is computed once from the training distribution and never changes. The robot stays just as cautious after a minute of driving, when it has effectively measured the floor's friction, as it was at power-on. We propose to make that extra cost depend on what the robot has observed so far. The robot keeps a running probability distribution over which randomized simulators match its own recent experience, and the penalty is the disagreement among the simulators that remain plausible. As evidence accumulates, the penalty shrinks in regions the robot has resolved and stays large where it has not.

This one change has a second effect. Once the penalty falls as uncertainty falls, a constrained learner has a direct incentive to take small, cheap, safe actions that reveal the hidden physics, such as a gentle brake test or a slow first stride. Information gathering then comes out of the safety constraint itself, with no separate curiosity bonus. The risk is that the robot becomes confident in the wrong simulator, especially when reality lies outside the randomized range. So the proposal adds an explicit check: when no simulator explains the observed transitions, the penalty reverts to its prior level or higher.

Core Claim

SPiDR supplies the starting method and the strongest result in this area. It keeps an ensemble of randomized dynamics, uses the variance of their next-state predictions $\upsilon(s,a)$ as a pessimistic surcharge on the safety cost, $\tilde c(s,a) = c(s,a) + \lambda\,\upsilon(s,a)$, and proves this upper-bounds the real cost. On hardware it satisfied constraints in every trial, while standard randomization produced 26.1x more violations on a Unitree Go1 and 6.2x more on a RaceCar. The limitation that matters here is that $\upsilon$ depends only on the current state and action, never on the history.

Our claim is that a belief-weighted surcharge keeps SPiDR's zero-shot safety and recovers much of the return SPiDR gives up to caution, when hidden parameters can be identified from a few safe transitions. It does so by inducing safe probing behavior that neither SPiDR nor an unconstrained belief-conditioned policy produces. Concretely, the surcharge is

$$\tilde c(s,a,b_t) = c(s,a) + \lambda \,\mathrm{Var}_{\xi \sim b_t}\big[f_\xi(s,a)\big] + \lambda \,\kappa(b_t)\,\upsilon_0(s,a),$$

where each term is defined as follows. The belief $b_t$ is a posterior over the $K$ ensemble members, each indexed by its simulator parameters $\xi$. Each member $f_\xi$ predicts the next state. The belief is updated by the likelihood of the transitions observed so far. The quantity $\upsilon_0$ is SPiDR's prior variance. The term $\kappa(b_t)\in[0,1]$ is a misspecification alarm: it rises toward 1 when even the best-fitting member explains recent transitions poorly, which is a predictive-likelihood test.

When $\kappa=0$ and the posterior equals the prior, the surcharge reduces exactly to SPiDR. The policy and the cost critic both take $(s, b_t)$ as input, summarized by the posterior weights or their mean and spread. This turns the problem into a constrained MDP over belief-augmented states.

Relation to Prior Work

Belief-conditioned control has two close predecessors. Uncertainty-Sensitive Privileged Learning trains an encoder that outputs a mean and a standard deviation over hidden parameters and conditions the expert on that spread. This produced active calibration behaviors and over 95% success on nine IsaacGym tasks, where RMA-style baselines sat near 35% or lower. Its exploration is driven by reward reshaping with no cost budget, and its authors note that agents can become trapped in cautious modes when ambiguity cannot be resolved. Robust Policy Learning over Multiple Uncertainty Sets updates a posterior uncertainty set every step and optimizes a risk-averse return (CVaR) over it. It does not impose a hard cost constraint, and the agent never acts to shrink the set. Both papers motivate our design: explicit uncertainty in the state helps. Both leave the step we take open, which is to have the uncertainty set the safety cost.

Three lines of work constrain the design.

  • Discrete formal guarantees. Constrained Bayesian RL via Approximate Linear Programming and Risk-Averse Bayes-Adaptive RL solve constrained or risk-averse problems exactly over beliefs, but only in small discrete domains. Provably Safe Sim-to-Real Transfer certifies zero real violations by shrinking a simulator-mismatch region, but it is tabular, uses a single nominal simulator, and needs a known safe baseline policy. We give up formal guarantees of this kind and aim for continuous control with domain randomization.
  • Shields. Adaptive Probabilistic Shielding shows that a conservative shield blocks exactly the actions needed to learn, and it resorts to unshielded exploration. Shielding under Partial Observability tracks the set of states consistent with history but can deadlock. Our mechanism keeps every probing action under the current pessimistic cost, rather than switching exploration off and on.
  • Violation data. Feasible Dual Policy Iteration shows that safe learners starve their cost critics of violation data. We use its dual-policy idea only in simulation, where violations are free, to keep the belief-conditioned cost critic sharp.

Research Program

Training happens entirely in a JAX/MJX simulator, matching SPiDR's setup. Each episode works as follows.

  1. A hidden parameter $\xi^\ast$ is drawn from the randomization prior.
  2. The agent sees observations only.
  3. A particle filter over $K$ ensemble members (we test $K=8$ to $32$; SPiDR reports negligible overhead for $n\ge 8$) updates $b_t$ from observed transitions.
  4. A constrained learner (PPO-Lagrangian, and separately SAC-Lagrangian) optimizes return subject to the budget on the expected sum of $\tilde c$.

Two design choices need care. First, posterior tempering: likelihoods are raised to a power below 1 and the effective number of particles is kept above a floor. This prevents premature collapse, which is the main way the method could become less safe than SPiDR. Second, the alarm $\kappa$: we calibrate a threshold on held-out in-support simulators so that it fires rarely there, then test whether it fires on out-of-support dynamics.

At deployment, the robot runs the same filter. Ensemble members are either run as MJX rollouts on an onboard GPU, or distilled into small next-state networks if latency requires it. We will report which option was needed per platform.

The analysis has three parts. We extend SPiDR's upper-bound argument to the belief-weighted cost. The bound holds whenever the true parameter lies in the posterior's high-probability set, and we will state the resulting violation bound in terms of that coverage probability. We measure empirically how often coverage fails. We also characterize when the induced probing appears, by varying how cheaply the hidden parameter can be identified.

Baselines

The naive reference point is standard domain randomization with PPO-Lagrangian: a constrained learner trained across randomized simulators with no pessimism and a recurrent policy. It shows how much of the problem is solved by constraints alone. Safety Gymnasium reports that PPO-Lag cut violations by 98% relative to PPO on velocity tasks, so this baseline is not a straw man in simulation. SPiDR's hardware results show it fails on hardware.

The strongest existing method is SPiDR. Its reported numbers are 100% constraint satisfaction on hardware, with standard randomization showing 26.1x (Go1) and 6.2x (RaceCar) more violations. No reported return figure for SPiDR on RWRL or PointGoal2 was found on Lacuna, so we will rerun it from its description. We will run SPiDR with the same ensemble, the same simulator parameter ranges, the same learner, and the same number of environment steps. Its $\lambda$ will be tuned over the same grid and seed budget used for ours, with selection on held-out simulators. We beat it if, at equal or lower real violation rates, we obtain higher return on identifiable tasks, and if we do no worse in violations on unidentifiable and out-of-support tasks.

We include two belief-conditioned competitors. The first gives SPiDR a recurrent or belief input but keeps the prior penalty $\upsilon_0$. The second is a USPL-style agent that is uncertainty-conditioned but trained without a cost constraint. Together they separate "knowing your uncertainty" from "being charged according to it."

The key ablations are:

  • belief-weighted penalty without the alarm $\kappa$, which isolates misspecification handling;
  • belief-weighted penalty without tempering;
  • an oracle that knows $\xi^\ast$ and uses the true simulator with $\lambda=0$, which upper-bounds achievable return.

Why not just…?

Why not just give SPiDR's policy a memory? A recurrent SPiDR policy can identify the dynamics internally, but its cost $\lambda\upsilon_0(s,a)$ does not depend on history. The return-optimal constrained policy therefore remains equally cautious after identification, and nothing rewards it for probing. The ablation that feeds the belief to the policy while keeping the prior penalty tests this directly. If that ablation matches our return, the core claim is wrong and we will say so.

Why not just anneal $\lambda$ over deployment time? A time schedule may work well when identification reliably happens within a fixed window. That makes it a real threat on easy tasks. It is blind, though, to whether identification actually occurred. On the unidentifiable variant and the out-of-support variant described below, a schedule should relax caution while the uncertainty remains. We will tune the schedule on held-out simulators with the same budget as our method and report where it matches us and where it breaks.

Why not just use implicit history encoding with Lagrangian constraints, as in RMA? USPL's evidence suggests that history encoders without an explicit uncertainty signal do not learn information-gathering behavior: RMA-style methods reached roughly 35% or lower on its tasks and stayed below 52% on Active T-Maze. That evidence comes from unconstrained tasks. We include the constrained version as the naive baseline above. If it matches our results, the explicit belief is unnecessary.

Why not just widen or tune the randomization range, as BayRn does? Bayesian Domain Randomization tunes ranges using real rollouts, with about 700 real rollouts across 20 policies, and those exploratory trials have no safety constraint. Our setting forbids unsafe real trials. Widening ranges also increases $\upsilon_0$ everywhere, which makes SPiDR-style caution worse rather than better.

Proposed Experiments

In simulation, the setting is SPiDR's RWRL tasks (Cartpole, Walker, Humanoid) and Safety Gym PointGoal2 in MJX, with a held-out "reality" parameter set. We build three variants of each task.

  • Identifiable: a safety-relevant parameter such as friction or actuator gain is revealed by a short safe probe.
  • Unidentifiable: the parameter does not affect any observation before it affects safety. This tests USPL's exploration trap, and here the method should behave like SPiDR.
  • Out-of-support: reality lies outside the randomization prior. This tests whether $\kappa$ fires.

We report, over many held-out realities and seeds:

  • the real constraint-violation rate and cumulative cost;
  • episode return;
  • the time until the policy's speed or aggressiveness reaches 90% of the oracle's;
  • the posterior coverage rate;
  • alarm firing rates.

We also probe the robustness channels of Robust Gymnasium (sensor lag, motor noise) as additional out-of-support stresses.

On hardware, we follow SPiDR's RaceCar obstacle task and, if available, Go1 locomotion, using surfaces with friction near the prior's edge.

The decisive checks are three. First, on identifiable tasks the method should beat SPiDR's return at no higher violation rate, and the prior-penalty ablation should not. Second, on unidentifiable tasks it should match SPiDR rather than relaxing. Third, on out-of-support dynamics its violations should stay at or below SPiDR's only when $\kappa$ is enabled. Failing the first check means the belief adds nothing. Failing the second or third means the method is less safe than what it replaces.