A robot trained in simulation and deployed on real hardware faces two very different kinds of uncertainty. The first is uncertainty the simulator can never remove. The real robot's friction, mass and sensor lag are unknown, and a camera image often does not reveal where an occluded obstacle is. The second is uncertainty that simply reflects the learner not yet having seen enough simulated data, and more simulation would remove it. We propose to measure these two separately inside a learned visual world model and to treat them in opposite ways. The agent is cautious, or "pessimistic," about the first when checking safety constraints. It is drawn toward the second, or "optimistic," as an exploration bonus during simulated training, where mistakes cost nothing. The goal is a camera-driven policy that transfers to new physics on the first try without violating constraints and without the frozen, do-nothing behavior that uniform caution produces.
This matters because the most successful method for safe zero-shot transfer applies one undifferentiated pessimism. SPiDR adds to the safety cost a penalty proportional to how much an ensemble of randomized simulators disagrees about the next state. It reached 100% constraint satisfaction on a Unitree Go1 and a small RaceCar, but it works on low-dimensional robot states, and its authors note that too much pessimism freezes the agent. Moving to pixels breaks the clean separation SPiDR relied on. A learned latent model is uncertain almost everywhere that hazards are rare in the data, so a direct pixel-space version of SPiDR would penalize the learner's own ignorance as if it were real-world danger. Our claim is that a simple law-of-total-variance split of an ensemble's predictions removes this confusion. It does so better than tuning a single penalty weight.
Core Claim
Let $h$ be the observation history (camera frames and actions) and $a$ the action. We train $K$ world-model members. Member $k$ predicts a distribution over the future safety cost $C$ given $(h,a)$, using simulated data in which the physical and visual parameters $\theta$ are randomized each episode. Because many combinations of true state and $\theta$ produce the same image history, each member's predicted spread already absorbs both domain uncertainty and perceptual ambiguity. The disagreement between members reflects only too little data. The standard decomposition, used for deep ensembles, is
$$ \mathrm{Var}[C \mid h,a] \;=\; \underbrace{\tfrac{1}{K}\textstyle\sum_k \mathrm{Var}_k[C \mid h,a]}_{U_{\text{real}}:\ \text{irreducible from the robot's view}} \;+\; \underbrace{\mathrm{Var}_k\big(\mathbb{E}_k[C \mid h,a]\big)}_{U_{\text{epi}}:\ \text{removable by more simulation}} . $$
The policy $\pi$ then solves a constrained problem in which only $U_{\text{real}}$ hardens the constraint, while $U_{\text{epi}}$ is added to the reward:
$$ \max_\pi\ \mathbb{E}_\pi\Big[\textstyle\sum_t \gamma^t \big(r_t + \alpha\, U_{\text{epi}}(h_t,a_t)\big)\Big] \quad \text{s.t.}\quad \mathbb{E}_\pi\Big[\textstyle\sum_t \gamma^t \big(\bar c_t + \lambda \sqrt{U_{\text{real}}(h_t,a_t)}\big)\Big] \le \epsilon . $$
Here $r_t$ is task reward, $\bar c_t$ the ensemble-mean predicted cost, $\gamma$ the discount, $\epsilon$ the safety budget, and $\alpha,\lambda$ are scalar weights, with $\alpha$ annealed to zero over training. The epistemic term never loosens the constraint. Before deployment, a certification step rejects any policy whose visited states still carry $U_{\text{epi}}$ above a threshold measured on training-range simulators.
The idea differs from the closest work in specific ways. SPiDR penalizes disagreement across given simulators in true state space, and it has no exploration bonus and no perception term. PIGDreamer learns from pixels with privileged-state critics, but it never randomizes domains and uses a plain Lagrangian. DOPE and OptPess-LP prove that pairing reward optimism with cost pessimism prevents conservative learners from getting stuck, but only in tabular problems. We take that insight from DOPE and use it in a regime its theory does not cover. We are honest that one part of DOPE does not carry over. In simulation, exploration need not itself be safe, so we borrow its "optimism unfreezes pessimism" mechanism and not its training-time safety guarantee.
Research Program
The backbone is a DreamerV3-style latent world model of the kind PIGDreamer and SafeDreamer build on. Its encoder is regularized with image shifts as in DrQ. We attach $K$ lightweight heads that predict a distributional cost-to-go, using quantiles, from the shared latent. We also add one reward head. Training runs in massively parallel MuJoCo-JAX (MJX) environments, as in SPiDR's port of PointGoal2. The randomization covers both dynamics (mass, friction, actuator gain, latency), as in quadruped sim-to-real, and visuals (texture, lighting, camera pose), as in (CAD)$^2$RL.
Privileged simulator state plays two roles. It trains the heads, through representation alignment as in PIGDreamer. It also provides a diagnostic, because in simulation we can compute the true spread of cost over $\theta$ for a given history and check whether $U_{\text{real}}$ is calibrated against it.
The work has three parts. The first is an analysis study: on Safety-Gymnasium vision tasks, we measure how much of a lifted-SPiDR latent penalty is epistemic versus irreducible, and how this ratio changes with training. We predict that the epistemic share dominates early and near rare hazards. The second is the method above, including the certification step. The third is evaluation that reports the consecutive-violation metric EMCC from Revisiting Safe Exploration alongside average cost. That paper shows that equal average cost can hide very different runs of consecutive violations, and those runs are what break hardware.
Baselines
The naive reference point is pixel-based domain randomization with PPO-Lagrangian and DrQ augmentation, using the Safety Gymnasium SafePO implementation with a PID-controlled multiplier. On state-based velocity tasks, that benchmark reports PPO-Lag giving a 98% reduction in violations at the price of a 45% drop in reward. We use this as the floor and also as the evidence that plain Lagrangians trade away reward heavily. Beating it means matching or lowering cost on held-out physics while recovering most of that lost reward.
Two pixel-native safe learners trained with our randomization are run next: Safe SLAC and PIGDreamer. Safe SLAC reports keeping average cost below the budget of 25 on nearly all of the SG6 suite in a single, non-randomized simulator. PIGDreamer reports near-zero cost and a 136% reward gain over the next privileged method on the Guard benchmark. Neither paper reports results under domain shift. These baselines test whether randomization alone, added to good pixel safe RL, is enough.
The strongest competitor is SPiDR itself, adapted to pixels in the most favorable way. Its variance penalty is computed in privileged true state across $n\ge 8$ parallel randomized simulators, the asymmetric critic consumes that state, and the actor sees only pixels. SPiDR's reported hardware result, 100% constraint satisfaction with plain domain randomization showing 26.1× more violations on the Go1 and 6.2× more on the RaceCar, is the bar this line of work has set. For SPiDR on PointGoal2 and on the RWRL suite, no reported result was found on Lacuna, so we will rerun it ourselves.
Three ablations isolate the key ingredient. The first, "lifted SPiDR," penalizes total latent ensemble variance with no split. The second applies the split with $\alpha=0$, removing the optimism. The third keeps optimism but penalizes total variance.
Fairness rules apply to all methods. Every method gets the same simulator steps, the same randomization distribution, the same encoder and world-model size where applicable, and the same tuning budget of twelve configurations. Configurations are selected only on training-range simulators, never on test conditions. Results are averaged over ten seeds. For SPiDR and lifted SPiDR, $\lambda$ is swept across the full budget so that each is compared at its best frontier. Our method "wins" if its reward-versus-EMCC Pareto frontier on held-out physics dominates that of privileged SPiDR at matched violation rates.
Why not just…?
Why not just tune SPiDR's $\lambda$ lower to stop the freezing? This is the main threat. One weight scales both kinds of uncertainty, so lowering it to free the agent near unexplored regions also weakens the penalty where physics truly varies. We predict that on tasks with rare hazards, such as PointButton1 and PointPush1, no single $\lambda$ reaches our frontier. The full $\lambda$ sweep in the baselines settles this directly. If lifted SPiDR's best point matches ours, the split adds nothing, and we will report that.
Why not just compute SPiDR's penalty on privileged state and let a pixel actor follow an asymmetric critic? It is cheap and inherits SPiDR's hardware track record. However, PIGDreamer proves that a policy acting on partial observations underestimates risk when the safety estimate is tied to states it cannot see. Safe SLAC's collapse in PointPush1 when the box blocks the camera is a concrete case of this. A state-based penalty ignores perceptual ambiguity, whereas $U_{\text{real}}$ includes it by construction. The decisive test is the occlusion-heavy tasks. If privileged SPiDR's EMCC matches ours there, perception-aware pessimism is unnecessary.
Why not just use worst-case robust training over $\theta$? SPiDR's authors describe minimax robust RL as hard to scale to high-dimensional continuous systems. In the tabular setting, DOPE shows that purely conservative learners such as OptPess-LP stay in a linear-regret phase much longer. We include a domain-adversarial variant that trains on the worst-performing $\theta$ in each batch as a secondary baseline. Our expectation is that it freezes, but the experiment decides.
Proposed Experiments
The main testbed is Safety-Gymnasium's first-person vision versions of PointGoal2, CarGoal1, PointPush1 and PointButton1, plus Guard Ant navigation, all run in MJX with randomization. Each task is tested on three splits: training-range parameters, held-out parameters inside the training support, and parameters outside the support (for example friction beyond the training range, and unseen textures). The outside-support split is the condition SPiDR's assumption excludes and where our calibration may also fail. Metrics are task return, average cost against the budget, violation rate, and EMCC.
There are three decisive checks. The first is the Pareto comparison against privileged and lifted SPiDR under the full $\lambda$ sweep. The second is a freezing diagnostic: the fraction of held-out episodes in which the agent moves less than a fixed distance, which should fall with optimism and rise without it. The third is a calibration check, comparing $U_{\text{real}}$ against the true spread of cost across $\theta$ obtained from privileged simulator rollouts. If calibration fails outside the support, we report that as the method's limit rather than hiding it.
A physical run on a camera-equipped 1/10-scale RaceCar, mirroring SPiDR's platform, is the final stage. It is contingent on the simulated results. Collisions and EMCC there are measured against SPiDR's privileged pixel variant over a fixed number of laps.
References
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- PIGDreamer: Privileged Information Guided World Models for Safe Partially Observable Reinforcement Learning
- DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
- Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- Revisiting Safe Exploration in Safe Reinforcement learning
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representation