Robots trained in simulation usually fail on hardware for two reasons that compound. The physics is slightly wrong: friction, mass, latency and compliance differ from what the simulator assumed. The sensors also lie, adding drift, bursts and correlated noise that the simulator never produced. Current safe-transfer methods handle this by being pessimistic. They penalize any behavior the simulator is unsure about and then never revisit that judgment, which yields robots that are safe but often too timid to learn anything from reality. This note proposes that a robot use its first real-world episodes to find out which simulator is the right one, and to do so safely. The robot keeps a probability distribution over simulator settings, covering both physics and sensor noise. Before each decision it picks one plausible simulator at random from that distribution and acts as if that simulator were true. It executes the action only if the action stays within the safety budget across many plausible simulators at once.
This split is the core idea. Thompson Sampling, meaning "draw one plausible world from your beliefs and act greedily in it," supplies exploration: actions are mildly optimistic, and the optimism shrinks automatically as evidence accumulates. A pessimistic check over many posterior draws supplies safety. The sensor model sits inside the same distribution, so the safety check reasons about where the robot probably is, not about a raw noisy reading. If this works, safe transfer stops being a fixed, conservative zero-shot gamble and becomes an adaptation process. Its safety margins tighten exactly where real data justifies tightening them, and no faster.
Core Claim
Our claim is that randomized optimistic proposals, filtered by a posterior-quantile safety check that includes sensor-noise parameters, reach near-oracle task return in fewer real episodes than pessimism-only adaptation, with the same or fewer constraint violations. The underlying reason is the "safety paradox" identified in Breaking Safety Paradox with Feasible Dual Policy Iteration: an agent that strictly avoids constraint boundaries never collects the near-boundary data needed to locate them. Posterior sampling pushes toward the boundary only in proportion to remaining uncertainty. The quantile filter keeps those pushes inside the budget.
The closest prior work divides into three groups, each missing one ingredient.
The first group supplies the safety mechanism we generalize but stays pessimistic and static. SPiDR adds a simulator-ensemble disagreement penalty to the cost and reports that it "satisfied safety constraints 100% of the time" on hardware. It is purely zero-shot, though: its penalty weight is hand-tuned, and it can freeze the robot. It also computes disagreement at the observed state, which assumes the observation is accurate.
The second group supplies the problem formulation and constrains our ambitions. Provably Safe Sim-to-Real Transfer shows zero-violation adaptation whose real-sample cost scales with the size of the mismatch region. Its guarantees, however, are tabular, explored with a safe-mode mixture policy, and depend on a separability assumption: the simulator must be either exactly right or wrong by a detectable margin. Smooth friction drift or sensor noise breaks that assumption.
The third group supplies the method. Safe Linear Thompson Sampling shows that posterior draws can expand a certified safe set with no penalty to the regret rate, but only for linear bandits and with variance inflation applied at every round. Learning Active Task-Oriented Exploration Policies shows that probing only the task-relevant physics helps (14g versus 22g pouring error). It explores without any safety constraint and collapses its belief to a point estimate.
Our proposal differs from all of them in four ways at once. It runs in continuous control. It adapts online rather than zero-shot. It keeps a full posterior that includes sensor parameters. It treats Thompson Sampling as the exploration engine inside a pessimistic filter, not as a replacement for that filter.
Research Program
We first train two components in simulation, both conditioned on a simulator parameter vector $\theta = (\phi, \psi)$. Here $\phi$ holds physics parameters (masses, friction, latency) and $\psi$ holds sensor-noise parameters (bias, variance, correlation time, burst rate). The first component is a policy $\pi(a \mid b, \theta)$ that acts from a state estimate $b$. The second is a cost critic $Q_c(s, a; \theta)$ that predicts discounted future constraint cost. Both are trained across the domain-randomization range, so that no retraining is needed online when the belief about $\theta$ changes. A backup policy is also trained; following the shielding design of Sim-to-Lab-to-Real, it is used as a fallback, with its own reachability-style critic.
On the real system, we maintain a particle posterior $p(\theta \mid D_t)$ over the data $D_t$ collected so far. Particles are reweighted by the likelihood of observed real transitions under each particle's simulator, including that particle's sensor model, and are rejuvenated with small perturbations. Because each particle carries its own sensor model, it also carries its own state filter. This gives a per-particle belief over the true state rather than a single noisy reading.
At each step the controller samples one particle $\tilde\theta_t$ and proposes $\tilde a_t = \pi(b_t^{\tilde\theta_t}, \tilde\theta_t)$. It accepts the proposal only if
$$\text{Quantile}_{1-\delta}\Big[\, Q_c(s, \tilde a_t; \theta^{(k)}) \;:\; \theta^{(k)} \sim p(\theta \mid D_t),\; s \sim b_t^{\theta^{(k)}} \Big] \le \epsilon_t,$$
where $\delta$ is the tolerated risk level, $\theta^{(k)}$ are $K$ posterior particles, $b_t^{\theta^{(k)}}$ is that particle's state belief, and $\epsilon_t$ is the remaining safety budget. If the check fails, the backup policy acts. Optimism therefore lives only in the proposal and pessimism only in the acceptance test. As the posterior concentrates, both shrink together, with no penalty weight to schedule.
We will also study an inflation knob suggested by the limitation of Safe-LTS. Proposals are sampled from a tempered posterior whose temperature rises only when recent proposals are being rejected near the boundary. This tests whether adaptive inflation recovers boundary expansion without the constant over-exploration that the fixed-inflation scheme induces.
On the analysis side, we aim for a modest but honest result. In the linear-Gaussian special case, linear-quadratic control with a Gaussian sensor, where Thompson Sampling analysis already exists (Thompson Sampling for Linear-Quadratic Control Problems), we will try to show that the quantile filter preserves stage-wise safety with probability at least $1-\delta$ per step under a well-specified posterior. We will not claim guarantees beyond that case.
Baselines
The naive reference point is standard domain randomization: a policy trained across the same parameter range, with no safety cost and no adaptation. The file reports its weakness directly. SPiDR found that standard domain randomization produced "26.1x higher safety violations" on the Unitree Go1 and "6.2x higher" on the RaceCar.
The strongest existing method is SPiDR itself, using the same simulator ensemble, the same constrained learner (PPO-Lagrangian, which Safety-Gymnasium reports cut velocity-task violations by 98% relative to PPO), and its penalty weight tuned with the same simulation-side validation budget we give our own $\delta$. SPiDR is the right competitor because it is the simplest method in the file that achieves real-world constraint satisfaction under physics mismatch. Its 100% satisfaction figure is reported on its own hardware, so we compare on its open MJX tasks rather than quoting the number across testbeds.
We will also run "SPiDR with refit," which re-estimates the ensemble from the same real data our posterior sees and retrains or reweights accordingly. This baseline isolates whether online adaptation alone, without randomized proposals, explains any gain.
From the active-identification side we include a point-estimate pipeline in the spirit of Learning Active Task-Oriented Exploration Policies. It estimates $\hat\theta$ from real data, conditions the same policy on $\hat\theta$, and protects it with the same backup shield, so that only the posterior uncertainty is removed.
On the tabular windy gridworld we run the mixture-policy method of Provably Safe Sim-to-Real Transfer. It reports "0% unsafe iterations" and needs 77,958 real samples against 296,910 for a no-simulator baseline.
Three ablations isolate our ingredients. The first replaces the Thompson proposal with the posterior-mean action. The second removes $\psi$ from the posterior and feeds raw observations to the filter. The third fixes the inflation temperature.
All methods receive the same simulator steps, the same real-episode budget, the same network sizes and the same number of hyperparameter trials.
Beating the strongest baseline means two things together. Cumulative real constraint violations must be no higher than SPiDR-with-refit; Expected Maximum Consecutive Cost from Revisiting Safe Exploration must also be no higher, since it penalizes long violation streaks that average cost hides. And the method must reach 90% of an oracle policy's return (one trained with the true parameters) in significantly fewer real episodes across seeds.
Why not just…?
Why not just use SPiDR and refit the ensemble as real data arrives? This is the real threat. If pessimism with an updated ensemble already shrinks the penalty fast enough, Thompson Sampling adds nothing. The safety-paradox argument predicts that a pessimism-only learner rarely visits near-boundary states, so its refit data stays uninformative exactly where it matters. The SPiDR-with-refit baseline and the posterior-mean ablation test this directly. If either matches our time-to-90%-oracle within noise, we will conclude the randomized proposal is unnecessary and report the refit method as the contribution.
Why not just estimate the parameters once and shield the resulting policy? Point estimates discard the uncertainty that both the safety check and exploration need. Safe-LTS shows that non-randomized safe exploration (Naive Safe-LUCB) can stall in suboptimal safe regions with linear regret. The point-estimate baseline measures whether that stalling appears in continuous control. The 14g-versus-22g result shows task-focused identification matters, but it was obtained without any safety layer.
Why not just add sensor-noise augmentation or an autoencoder in simulation and leave the safety filter alone? Augmentation helps when real noise resembles the simulated noise. The file notes that filters trained on synthetic noise often fail on correlated or bursty artifacts, and Blind Spot Detection shows that compressed representations can be confidently wrong about safety-relevant cues. Our no-$\psi$ ablation is this alternative, run with the same augmentation. The comparison under bursty, correlated noise will show whether inferring the sensor model online changes false shield triggers and missed violations.
Proposed Experiments
The main suite uses SPiDR's MJX ports of RWRL Cartpole, Walker and Humanoid and Safety Gym PointGoal2. In each, "reality" is a held-out simulator whose physics comes from RRLS-style ModifiedParamsEnv shifts in mass, friction and damping (RRLS). Observations in that held-out simulator are corrupted by one of three sensor regimes: independent Gaussian noise, slow bias drift, or correlated bursts. Crucially, half the shifts are chosen to be small and smooth, so they violate the separability assumption.
For Safety-Gymnasium PointGoal1 and Circle tasks we additionally report Expected Maximum Consecutive Cost. On the 5x5 windy gridworld we check that our method approaches the 0% unsafe-iteration and 77,958-sample reference reported for the tabular method. That check confirms nothing is lost in the setting where guarantees exist.
Three decisive checks follow, each tied to a specific ingredient.
The first concerns the randomized proposal. On shifts where the safe region in reality is larger than the simulator ensemble believes, our method should recover the extra return while SPiDR-with-refit stays frozen. If it does not, the core claim fails.
The second concerns the sensor posterior. Under bursty noise, the method with sensor parameters in the posterior should show fewer false backup-policy interventions and fewer missed violations than the ablation without them. If the two match, modeling sensor noise inside the posterior is unjustified.
The third concerns support. When reality lies outside the randomization range, the posterior should collapse onto boundary particles and the quantile test should trigger the backup policy persistently. We will report how often violations occur in that regime rather than hide it, since SPiDR shares this support assumption.
If simulation results hold, a final small hardware study will follow SPiDR's 1/10th-scale RaceCar protocol: obstacle avoidance under latency and friction shifts, with deliberately degraded range sensing. It will measure violations and real episodes to reach target lap performance.
References
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Provably Safe Sim-to-Real Transfer
- Safe Linear Thompson Sampling
- Learning Active Task-Oriented Exploration Policies for Bridging the Sim-to-Real Gap
- Sim-to-Lab-to-Real: Safe RL with Shielding and Generalization Guarantees
- Thompson Sampling for Linear-Quadratic Control Problems
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Revisiting Safe Exploration in Safe Reinforcement learning
- Blind Spot Detection for Safe Sim-to-Real Transfer
- Rrls: Robust reinforcement learning suite