Chance-Constrained Gaussian Policies for Safe Sim-to-Real Transfer

Robots trained in simulation usually act by sampling from a Gaussian over actions. The spread of that Gaussian is what lets them explore, and it is also what makes them dangerous. Existing safety layers, which are small modules that nudge a proposed action back inside a safe region, only move the center of the Gaussian. The random draw around that center can still cross the boundary. On a real robot a second source of risk sits on top of this one: the simulator's physics is wrong by an unknown amount. We propose one safety layer that handles both. It reshapes the whole action distribution, center and spread together, so that the chance of violating a constraint on the next step stays below a user-chosen probability $\delta$. The physics uncertainty enters through the disagreement among an ensemble of randomized simulators.

The layer has two practical effects. It replaces the hand-tuned penalty weight that current zero-shot safe-transfer methods use with a single interpretable number, the tolerated per-step violation probability. It also lets the policy keep full randomness in directions that cannot cause harm, instead of shrinking all of its exploration uniformly. If this works, safety constraints can be trained in simulation and held during real-world fine-tuning, where random exploration is unavoidable, without the robot freezing into "safe but useless" behavior.

Core Claim

Two lines of work each solve half of this problem. Safe Policy Learning for Continuous Control projects the policy's mean action onto a linearized safe half-space. It moved a 150 kg Fetch robot "over 500 meters of safe travel in a dynamic office", but its authors state that the projection leaves the variance unconstrained, so sampled actions can still violate the constraint. SPiDR makes a policy pessimistic about reality by adding the variance of an ensemble of randomized dynamics models to the cost, $\tilde c = c + \lambda \upsilon$. With that it reaches "100% constraint satisfaction in the real world" on a Unitree Go1. SPiDR has three limits for our purposes: the weight $\lambda$ is heuristic and must be tuned, it assumes reality lies inside the randomization support, and it never looks at the policy's own action spread. Covariance Steering for Systems Subject to Unknown Parameters shows how to turn linear chance constraints under parametric uncertainty into tractable convex constraints using Cantelli's inequality. It does this for trajectory optimization, not for deep policies. Our claim is that the Cantelli construction can be applied per step to a Gaussian policy. The uncertainty term then combines policy randomness and simulator disagreement, and the result is a differentiable layer that constrains the full distribution with a calibrated margin.

Concretely, suppose that at state $s$ the next-step constraint is linearized as $c(s) + g^\top a \le b$. The sensitivity vector $g$ differs across the $K$ randomized simulators in the ensemble, with mean $\bar g$ and covariance $\Gamma$. The policy proposes $a \sim \mathcal N(\mu, \Sigma)$. Cantelli's inequality gives the one-sided bound $P(\text{violation}) \le \delta$ whenever

$$\bar g^\top \mu + \kappa_\delta \sqrt{\bar g^\top \Sigma \bar g + \mu^\top \Gamma \mu + \operatorname{tr}(\Gamma\Sigma)} \le b - c(s), \qquad \kappa_\delta = \sqrt{(1-\delta)/\delta},$$

where the square-root term is the standard deviation of $g^\top a$ when both $g$ and $a$ are random. The layer outputs the Gaussian $\mathcal N(\mu', \Sigma')$ closest in KL divergence to the proposal that satisfies this inequality. With one active constraint, the optimal solution shifts the mean along $\bar g$ and shrinks the covariance only along $\bar g$ and the directions exposed by $\Gamma$. The spread orthogonal to the risky directions is left intact, and the solution reduces to a one-dimensional root search. With several constraints the problem is a small second-order cone program; differentiable convex layers can solve it in batch.

This has two consequences. First, the scalar $\lambda$ of SPiDR is replaced by $\delta$, which has a meaning: the tolerated violation rate per step. Second, setting $\Sigma = 0$ and $\Gamma = 0$ recovers the mean-only projection of Chow et al. as a special case, so the contribution of variance shaping and of the dynamics term can each be measured by ablation.

Research Program

We will first build the layer on top of a Gaussian-policy actor-critic, starting with a proximal policy optimization agent. The constraint sensitivity $g$ comes from a learned one-step cost model. The model has $K$ heads, and each head is trained on data from a different block of the domain-randomization distribution. $\bar g$ and $\Gamma$ are therefore empirical moments over simulators, the same disagreement signal SPiDR uses, but applied to the constraint's dependence on the action rather than added as a flat penalty. Log-probabilities for the policy gradient are computed under the projected Gaussian, which is well defined because the output is still a Gaussian. Per-sample clipping lacks this property, since it piles probability mass onto the boundary.

We then study three questions. The first is calibration: across held-out simulator parameters, does the empirical per-step violation frequency track $\delta$? The Cantelli bound is distribution-free, so we expect over-conservatism rather than under-coverage whenever reality lies inside the ensemble's support. The second is the out-of-support case that Safe Domain Randomization via Uncertainty-Aware Out-of-Distribution Detection and Policy Adaptation argues is common. Here we measure how the guarantee degrades once test physics leave the randomization range, and whether $\Gamma$ inflating on novel states gives an early warning. The third is the freezing trade-off: does preserving orthogonal randomness keep task reward above that of uniformly conservative methods at the same violation rate?

The layer constrains only the next step. Long-horizon feasibility is handled by the constraint signal $c(s)$. Following Constrained Markov Decision Processes via Backward Value Functions, which converts trajectory budgets into state-wise constraints, we set $c(s)$ as cost-so-far plus a learned cost-to-go. This role is borrowed from that paper, not claimed as new. We also note the safety paradox identified in Breaking Safety Paradox with Feasible Dual Policy Iteration: as a policy becomes safer, it stops visiting unsafe states, and its model of the safety boundary degrades from lack of data. Our layer does not fix this. It deliberately operates in simulation during training and keeps $\delta$ small in the real world. In simulation, the ensemble disagreement stays informative because boundary data can be generated freely.

Baselines

The naive reference is standard domain randomization with an unconstrained Gaussian policy, the setting in which SPiDR reports "26.1x higher safety violations" on the Go1 and "6.2x higher" on the RaceCar than its own method. A second reference is PPO-Lagrangian trained under the same randomization. Safety Gymnasium reports that PPO-Lag gives "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward" on velocity tasks. This tells us what reward cost a plain Lagrangian pays. Beating it means matching its violation rate with a clearly smaller reward drop.

The strongest competitor is SPiDR itself. It targets exactly the same zero-shot safe-transfer question, uses the same ensemble disagreement signal, and has hardware evidence. We will reimplement it on the MJX port of Safety Gym that its authors describe. It will use the same randomization distribution, the same ensemble size, the same network sizes and the same number of environment steps. $\lambda$ gets a grid search with the same number of trials we give to $\delta$. Beating SPiDR means Pareto dominance on the reward-versus-violation curve under held-out physics. It also means one fixed $\delta$ transferring across tasks, where SPiDR's best $\lambda$ does not. Because SPiDR reports 100% constraint satisfaction on hardware, we do not expect to win on raw safety inside the support. The contest is reward at matched safety and robustness of the tuning.

Three ablations isolate the key ingredients. Mean-only projection with the Cantelli margin keeps $\Gamma$ but does not reshape $\Sigma$, and is the Chow et al. layer with a dynamics-aware margin. Full distribution projection with $\Gamma = 0$ reshapes the spread but ignores simulator disagreement. The third ablation replaces Cantelli with a Gaussian quantile, testing whether the distribution-free bound is too loose to be useful.

Why not just…?

Why not just project each sampled action onto a robust, worst-case-over-the-ensemble half-space? This is the most serious threat. It is cheap, it is safe per sample under the ensemble, and it needs no change to the policy distribution. It has two weaknesses. Taking the worst case over $K$ heads grows more conservative as $K$ grows and has no calibrated knob. Clipping samples also leaves the policy gradient computed under an unprojected distribution, so the policy never learns to move its own spread away from the boundary. We will run this baseline with identical ensembles. If it matches our reward at equal violation rates, the honest conclusion is that the contribution shrinks to the calibrated $\kappa_\delta$ margin, and we will report it that way.

Why not just tune SPiDR's $\lambda$ per task? On a fixed task with generous tuning, a well-tuned $\lambda$ may match us. Our experiment gives both methods the same tuning budget and then tests transfer of the chosen hyperparameter to new tasks and to shifted physics. SPiDR's own paper documents the need for manual tuning, and the question is whether $\delta$ removes it.

Why not just deploy the deterministic mean action and skip variance control? For zero-shot deployment alone, this is reasonable. In that case our method reduces to a mean shift with a disagreement-aware margin, which the first ablation measures directly. The variance term matters once the robot must keep exploring on hardware during fine-tuning. That is the regime Safe Exploration via Policy Priors targets, and it needs a safe prior policy and resets that we do not assume. We therefore separate zero-shot and fine-tuning results rather than claim a benefit where the variance plays no role.

Why not just use a Lagrangian multiplier? The evidence in the file is against it as a per-step guarantee. LyEvO reports violation rates up to 54% for SAC-Lagrangian in simulation rollouts. Chow et al. report that the multiplier oscillates because it reacts slowly to cost spikes.

Proposed Experiments

The main testbed is Safety-Gymnasium's Point and Car goal tasks and its velocity-limited MuJoCo tasks, run through the MJX Safety Gym port used by SPiDR. We randomize mass, friction, actuator gain and latency, which covers the actuator and contact sources of the reality gap. For each method we train under a nominal randomization range. We then evaluate on physics drawn from inside the range, from its edges, and from outside it at a 1.5× and 2× widening. Metrics are episode return, cumulative constraint cost, and the empirical per-step violation frequency plotted against $\delta$ for $\delta \in \{0.01, 0.05, 0.1\}$. On the reward-versus-violation curve, the distance to SPiDR is the headline number.

For hardware, we will use a 1/10-scale race car on obstacle avoidance, the platform on which SPiDR reports 6.2× fewer violations than DR and SOOPER reports a policy "twice as good as the initial safe prior" with no collisions. The protocol is zero-shot deployment followed by a short fine-tuning phase on hardware with stochastic exploration left on, run under the robust-projection baseline, SPiDR and our layer.

Three results would decide the question. First, the calibration curve must stay at or below the diagonal inside the support; if it does not, the construction fails. Second, at matched violation rates, full distribution projection must beat mean-only projection on return, since otherwise variance shaping adds nothing. Third, a single $\delta$ must hold across tasks where SPiDR needs retuning of $\lambda$. A negative outcome on the out-of-support tests is expected and informative. It tells us whether the growth of $\Gamma$ can serve as the task-relevant shift detector that Task-Driven Detection of Distribution Shifts With Statistical Guarantees for Robot Learning builds offline with PAC-Bayes bounds, here obtained online at no extra cost.