Robots trained in simulation face two kinds of change when they reach the real world. The first is a change of physics: friction, mass and latency differ from the simulator. The second is a change of job: a new goal to reach, a new obstacle to avoid, a new speed limit. Current methods handle one change or the other, never both. We propose to learn, in simulation, how a small library of behaviors accumulates measurable quantities (distance to hazards, speed, contact, progress) under many randomized versions of the physics. Once that is stored, any new reward or safety cost that is a weighted sum of those quantities can be evaluated at deployment by taking dot products. The safety margin can then be computed for the specific cost at hand, as a pessimistic quantile over the simulated physics, instead of being a fixed penalty on "places where simulators disagree."
The practical payoff is a controller that can be given a new task and a new safety budget at deployment and satisfy that budget on unseen physics without any retraining. It should also be less timid than existing pessimistic sim-to-real methods, because it only pays for uncertainty that actually affects the cost being constrained.
Core Claim
Successor features are the expected discounted sum of a feature vector $\phi(s,a,s')\in\mathbb{R}^d$ under a policy. They let a value function factor into a task-independent part and a task weight: $Q^\pi = \psi^\pi(s,a)^\top w$ (Successor Features for Transfer in Reinforcement Learning.). That factorization assumes fixed dynamics. Domain randomization, which trains under perturbed simulator parameters $\theta$ so the policy survives the real world, breaks the assumption on purpose.
We turn this conflict into a tool. We learn physics-conditioned successor features $\psi^{\pi_i}(s,a,\theta)$ for each library policy $\pi_i$, where each policy is itself trained with ordinary domain randomization. For any cost weight $w_c$ given at deployment, we then define a cost bound that depends on the direction of $w_c$:
$$C^{+}_i(s,a;w_c)=\mathrm{Quantile}_{1-\delta}\big[\psi^{\pi_i}(s,a,\theta)^\top w_c\big],\quad \theta\sim P(\Theta),$$
where $\delta$ is the tolerated probability of exceeding the bound. Reward is scored by the mean $\bar R_i(s,a;w_r)=\mathbb{E}_\theta[\psi^{\pi_i}(s,a,\theta)]^\top w_r$.
Actions are chosen by a constrained version of generalized policy improvement, meaning the agent acts greedily over the whole library:
$$a^\ast=\arg\max_a\max_i\big[\bar R_i(s,a;w_r)-\lambda\,C^{+}_i(s,a;w_c)\big].$$
Here $\lambda$ is a Lagrange multiplier, the weight the agent currently places on the cost. It is solved per task from the library's values at start states.
The central claim is about where the pessimism goes. Uncertainty across $\theta$ that lies in feature directions orthogonal to $w_c$ contributes nothing to $C^+$. A cost about hazard contact is therefore not inflated by uncertainty about forward speed, and a speed-limit cost is not inflated by uncertainty about lateral drift.
Relation to Prior Work
Two papers motivate the gap from opposite sides. Safety-Constrained Policy Transfer with Successor Features (SFT-COP) factors both reward and cost into successor features and solves a Lagrangian dual over source policies. This gives safe transfer across reward and cost weights, but only under fixed, nominal dynamics with discretized actions. SPiDR adds the variance of next-state predictions across randomized simulators to the safety cost. It reports that plain randomization produced 26.1x more violations on a Unitree Go1 quadruped, and that SPiDR satisfied constraints 100% of the time. However, it is tied to one reward, its penalty weight is hand-tuned, and its penalty is isotropic: it treats disagreement in any state dimension as risk.
We keep SFT-COP's dual machinery and SPiDR's use of the randomization ensemble as a source of pessimism. We change what the pessimism is computed over, moving from one-step state variance to the multi-step, cost-projected spread of successor features.
Two further papers supply method. Constrained GPI shows that universal successor-feature networks overestimate values on task vectors far from training, and it clips predictions to provable bounds from linear combinations of source tasks. We apply the same clipping to both $\bar R$ and $C^+$ before the quantile is taken, so that neural extrapolation errors do not masquerade as physics uncertainty. Distributional Successor Features models distributions over successor features for zero-shot optimization. Our distribution is over physics for a fixed policy, not over reachable outcomes in a fixed world, and we do not need its diffusion sampler.
One paper constrains the design. Safe Domain Randomization via Uncertainty-Aware OOD Detection reports that broad randomization produces slow, jerky, over-conservative gaits. This warns that a worst-case maximum over $\theta$ would reproduce that failure, which is why we use a quantile. Provably Safe Sim-to-Real Transfer gives tabular guarantees in which real-world samples scale with the size of the mismatch region. We make no such guarantee; our contribution is empirical and continuous.
Research Program
We will build four components.
The first is a feature map $\phi$ for Safety-Gymnasium-style navigation and locomotion. We will hand-design it where possible (hazard proximity indicators, contact, velocity components, goal progress, action magnitude) and, as a secondary study, learn it by regression on a fixed bank of reward and cost signals. Linearity of costs in $\phi$ is the method's main assumption. We will measure how much cost variance the chosen $\phi$ fails to explain on held-out costs.
The second is a library of $n\approx 8$–$16$ policies. Each is trained with domain randomization and a Lagrangian method (PPO-Lag from SafePO in Safety Gymnasium) on a different source pair $(w_r,w_c)$.
The third is a physics-conditioned successor-feature network per policy. It will be trained by temporal-difference learning on rollouts in which $\theta$ is sampled per episode and fed to the network as input. $\theta$ is available to this evaluator only, never to the policy, in the spirit of the privileged critic in Asymmetric Actor Critic.
The fourth is the deployment rule. At deployment the real $\theta$ is unknown, so $C^+$ is computed by Monte Carlo over $\theta$ samples, CGPI clipping is applied, and $\lambda$ is set with the SFT-COP dual.
The analysis side asks three things. When does direction-aware pessimism differ materially from isotropic pessimism? How well does the quantile level $\delta$ track the empirical violation rate on held-out physics? How does performance degrade when the test $\theta$ lies outside the randomization range?
Baselines
The naive reference is SFT-COP trained under plain domain randomization, with successor features averaged over $\theta$ and costs scored by the mean. This isolates whether any pessimism is needed. The literature suggests it will be unsafe: SPiDR's 26.1x (Go1) and 6.2x (RaceCar) violation ratios for plain randomization are the reason.
The strongest existing competitor is SPiDR retrained separately for every test task, with its variance-penalized cost and a tuned $\lambda$. SPiDR is the only method in the file with demonstrated zero-shot sim-to-real constraint satisfaction. Its per-task retraining makes it an upper bound on what monolithic training can achieve. It gets the same simulator, the same randomization distribution, a per-task environment-step budget reported alongside, and the same hyperparameter-search budget as our method, including a sweep over its penalty weight. The only reported numbers on Lacuna for it are on hardware (100% constraint satisfaction on Go1 and RaceCar; 26.1x and 6.2x fewer violations than plain randomization). For PointGoal2 in MJX we found no reported numeric result on Lacuna, so we will reproduce it. We beat it if, at matched or lower violation rates on held-out physics, our zero-shot controller reaches equal or higher reward on held-out tasks without any per-task training. Losing on reward while winning on training cost counts as a partial result, and we will report it that way.
The key ablation replaces $C^+$ with SFT-COP plus a SPiDR-style isotropic penalty: mean cost plus a multiple of the trace of successor-feature variance across $\theta$. This keeps everything identical except the direction of the pessimism.
A second ablation removes CGPI clipping. It measures how much apparent physics uncertainty is really network extrapolation error.
For context we also report PPO-Lag trained from scratch per task, whose reported profile is a 98% reduction in violations and a 45% drop in reward relative to PPO in Safety Gymnasium.
Why not just…?
Why not just retrain SPiDR per task, since GPU simulation is cheap? This is the real threat. With MJX, per-task retraining may take minutes, and a tuned per-task policy may simply score more reward. Our answer has two parts, and the experiments settle both. First, we report the reward and violation gap at equal wall-clock and step budgets, including a near-zero budget. Second, we test whether a single isotropic penalty weight can be tuned to be safe across many cost directions. If per-task SPiDR dominates at small retraining cost and one weight transfers well, then our contribution narrows to instant task switching, and we will say so.
Why not just penalize reward with $r-\beta c$? Scalar penalties are brittle in both directions, either accepting crashes or freezing (Quanyi Li et al.). A fixed $\beta$ also cannot adapt to a new budget. Our dual solve does this per task, and the penalty baseline run across budgets will show the difference.
Why not just use a worst case over $\theta$, or an ensemble of cost critics? A worst case over broad randomization yields the sluggish behavior reported for broad randomization in Safe Domain Randomization via Uncertainty-Aware OOD Detection. We sweep $\delta$ down to the worst-case limit to show that trade-off. A scalar cost-critic ensemble is exactly our method restricted to one fixed $w_c$. It is a fair competitor on source tasks and is inapplicable to new cost vectors without retraining.
Proposed Experiments
The main testbed is Safety-Gymnasium PointGoal2 and CarGoal ported to MJX following SPiDR, plus a velocity-constrained locomotion task, with randomized friction, mass, actuator gain and latency. Held-out tasks change reward weights and introduce cost vectors not seen in training, such as reweighted hazard types and new speed caps. Held-out physics includes parameters both inside and just outside the randomization support. We measure return, cost relative to budget, the violation rate over episodes, and calibration of $\delta$ against the observed exceedance frequency.
The decisive check is a controlled randomization design. In one condition, the physics parameters mainly perturb cost-irrelevant features (for example, mass perturbing forward speed while the cost is hazard contact). In the other, they perturb exactly the costed features. Direction-aware pessimism should beat the isotropic ablation on reward in the first condition while matching it on safety, and the two should converge in the second. If they perform the same in both, the core claim is false.
A Go1 hardware test with a new cost weight given at deployment is a stretch goal. It would be run only after the simulated calibration results hold.
References
- Successor Features for Transfer in Reinforcement Learning.
- Safety-Constrained Policy Transfer with Successor Features
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- Constrained GPI for Zero-Shot Transfer in Reinforcement Learning
- Distributional Successor Features Enable Zero-Shot Policy Optimization
- Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation
- Provably Safe Sim-to-Real Transfer
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Asymmetric Actor Critic for Image-Based Robot Learning
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning