Which Seed Is Safe to Deploy? Certifying Individual Policies for Zero-Shot Safe Sim-to-Real Transfer

When a robot policy is trained only in simulation and then run directly on hardware, the transfer is called "zero-shot" sim-to-real transfer. A standard way to prepare policies for this is domain randomization: each training episode samples different physics (mass, friction, damping, motor delay) so the policy cannot depend on any single setting. Safety is usually handled with constrained reinforcement learning, which maximizes reward while keeping an expected safety cost under a budget. Recent methods that combine the two report perfect real-world constraint satisfaction. These methods leave something out, though. Deep reinforcement learning is notoriously sensitive to the random seed, and a practitioner does not deploy an algorithm. They deploy one trained network, from one seed. Nobody has measured how much the real-world safety of that network depends on which seed produced it, or whether one can tell, before touching hardware, which seeds are dangerous.

We propose to treat the training seed as a first-class random variable in safe sim-to-real transfer. We will measure it, decompose it, and gate deployment on it. Our claim is that seeds trained under the same randomization can all meet the safety budget on average while differing sharply in how their safety cost behaves on rare physics settings. We further claim that these differences can be detected cheaply in simulation by stress-testing each trained policy on physics deliberately pushed outside the training range. The result is a per-policy acceptance test with a statistical guarantee over the stress distribution. We will check empirically whether passing this test predicts safety on a held-out "reality."

Core Claim

Domain randomization addresses variation in the environment. Seed sensitivity is variation in the learner. The current safe-transfer literature evaluates only the first.

SPiDR adds a penalty to the safety cost wherever an ensemble of randomized dynamics models disagrees. It reports "100% constraint satisfaction in the real world," with standard randomization producing 26.1x more violations on a Unitree Go1. It does not quantify run-to-run variance. Its ensemble disagreement and its Lagrange multiplier (the adaptive weight on the safety cost) are both trained objects that can themselves depend on the seed. RRLS does run 10 seeds under randomized MuJoCo dynamics and finds severe seed variance in hard tasks such as Humanoid Stand Up, but it has no safety constraints.

We claim three things, and we test each. First, under safe domain randomization, a large share of the variance in held-out violation rate comes from the seed rather than from the dynamics sample. Second, the seed-level quantity that predicts transfer failure is not simulated return or mean cost. It is tail constraint cost on dynamics outside the training support. Third, a simple binomial acceptance test on that quantity gives a usable deployment rule. Under equal compute, this rule does better than spending the same budget on more pessimism, more parallel environments, or picking the seed with the best return.

The contrast with the closest work is direct. SPiDR and UARL change how a single policy is trained. We change which trained policy is allowed to ship, and we measure whether seeds differ enough for that choice to matter. Sim-to-Lab-to-Real also certifies policies before deployment. It uses PAC-Bayes bounds, which require "at least a few hundred 'Lab' environments" in the physical world. Our certificate is computed entirely on simulated stress environments, so it costs nothing in hardware time. It also promises less: it holds over the stress distribution, not over reality. Whether it transfers is exactly the empirical question we pose.

Research Program

The first component is a variance decomposition. For a training seed $s$, define its stress violation probability

$$p_s = \Pr_{\xi \sim \Xi_{\text{stress}}}\left[ C(\pi_s, \xi) > \epsilon \right],$$

where $\pi_s$ is the policy trained from seed $s$, $\xi$ is a vector of physics parameters, $\Xi_{\text{stress}}$ is a distribution over physics that extends beyond the training range, $C(\pi_s,\xi)$ is the episode's accumulated safety cost, and $\epsilon$ is the budget.

Across 30 seeds per method, the range recommended by Measuring the Reliability of Reinforcement Learning Algorithms, we split the variance of the per-episode violation indicator into two parts. One is variance in $p_s$ across seeds. The other is the within-seed variance across $\xi$ and rollouts. Following that same paper, we report dispersion across runs as the interquartile range, and tail risk as conditional value at risk (CVaR), the mean of the worst α-fraction of outcomes.

We also log three training-time signals that the literature suggests drive seed divergence. One is the trajectory of the Lagrange multiplier, which Safety Gymnasium shows oscillates. Another is the fraction of constraint-violating samples in the replay buffer, which falls in late training because of the "safety paradox" identified in Breaking Safety Paradox with Feasible Dual Policy Iteration. The third is activation saturation, which Is High Variance Unavoidable in RL? links to permanently stalled seeds. These signals let us say why bad seeds are bad, rather than only that they exist.

The second component is the acceptance test. For each trained seed we draw $n$ stress environments, count $k$ violating episodes, and accept the seed only if

$$\bar p_s(k, n, \beta / K) \le \delta,$$

where $\bar p_s$ is the one-sided Clopper–Pearson upper confidence bound on $p_s$ at confidence $1-\beta/K$, $K$ is the number of seeds tested (a Bonferroni correction, so the guarantee survives choosing among seeds), and $\delta$ is the tolerated violation probability.

Among accepted seeds we deploy the one with the highest return. This guarantee is exact over $\Xi_{\text{stress}}$ and silent about reality. Its value therefore depends on how $\Xi_{\text{stress}}$ is built. We compare two designs. The first is in-support stress, which is simply more samples from the training randomization. The second is beyond-support stress, which widens friction, mass, and damping ranges by fixed factors and injects actuator delay beyond the trained range. The delay injection follows Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, which found that modeling 15–19 ms of latency was necessary for transfer. The beyond-support design directly probes SPiDR's stated assumption that reality lies inside the randomized support.

The third component is the proxy-reality evaluation. Most labs cannot run 30 seeds times many trials on hardware. Our primary "reality" is therefore a held-out simulator configuration that no method sees during training or stress-testing: a different MuJoCo contact model, out-of-range latency, and unmodeled perturbations. Reality is a separate environment from the stress distribution, so a certificate can fail to transfer. If a 1/10-scale race car of the kind used by SPiDR is available, we will run a small hardware check on the accepted and rejected seeds with the most extreme scores. This check is confirmatory and is not where the statistics come from.

Baselines

The naive reference point deploys one randomly chosen seed of a Lagrangian PPO policy trained with standard domain randomization. This is what a typical paper's single-policy deployment amounts to. For context, Safety Gymnasium reports that on its velocity tasks "PPO-Lag achieved a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward." For this baseline we report the full across-seed distribution of proxy-reality violation rates, not a mean.

The strongest existing safe-transfer method is SPiDR. It is the only cited method that pairs randomized-dynamics pessimism with a constrained solver and reports hardware zero-shot safety. Its headline numbers are "100% constraint satisfaction in the real world" and 26.1x (Go1) and 6.2x (RaceCar) fewer violations than standard randomization. We reimplement it in MJX as the paper does, with an ensemble of at least 8 dynamics models, which the authors report adds negligible training time. We tune its pessimism weight on the same validation budget we give every other method.

Two optimization-side competitors attack seed variance at its source. The PID-Lagrangian variant (CPPO-PID) from Safety Gymnasium damps multiplier oscillation. FDPI keeps boundary data in the replay buffer and reports SAC-FDPI achieving "the lowest constraint violations (near-zero) across all 14 environments." If either method makes the across-seed spread small, selection becomes unnecessary. That outcome is a legitimate finding and not a failure of the study.

We found no reported result on Lacuna for the seed-level dispersion of real-world or held-out-dynamics violation rates for any of these methods. Establishing that number is part of the contribution.

Fairness is defined by compute. Selecting among $K$ seeds costs $K$ training runs plus stress evaluation. Every competitor therefore receives the same total environment steps, spent in whatever way best suits it. The options are one long run, one run with $K$ times as many parallel environments, or a $K$-member policy ensemble that averages actions. Hyperparameters for all methods are tuned on the same Safety-Gymnasium validation tasks with the same number of trials.

Three ablations isolate the key ingredient by changing the selection statistic while holding the seed pool fixed. The statistics are highest simulated return (the "seed lottery" that Deep Reinforcement Learning That Matters warns against), lowest mean cost on the training distribution, and CVaR on in-support stress. Beating the field means two things. At matched or better return, the accepted seed's proxy-reality violation rate must sit below the best competitor's median-seed rate. It must also be no worse than that competitor's 90th-percentile seed. Bootstrap confidence intervals over seeds must exclude zero difference.

Why not just…?

Why not just train many seeds and deploy the one with the best simulated return? This is the alternative our method most resembles, so the ablation must separate the two. Deep Reinforcement Learning That Matters shows that seed effects alone can make two identical TRPO setups look different at p = 0.0016. Measuring the Reliability of Reinforcement Learning Algorithms shows that median performance does not track tail risk. Selecting on return should therefore pick lucky seeds, not safe ones. Our ablation tests this directly. If return-based selection predicts proxy-reality safety as well as beyond-support CVaR does, the stress test adds nothing, and we will say so.

Why not just use a damped multiplier or many more parallel environments, so seed variance disappears? This is a real threat. With thousands of parallel environments, each update averages over many physics samples. That may wash out the early cluster of high-cost episodes that would otherwise spike one seed's multiplier, and CPPO-PID is designed to stop that oscillation. The compute-matched comparison settles the question. If CPPO-PID or a wide-parallelism run brings the across-seed IQR of proxy-reality violation rate near the within-seed noise floor, the decomposition will show it, and the proposal's main contribution becomes the negative result plus the measurement protocol. We expect much of the variance to survive. Activation saturation and buffer starvation act per seed and are not averaged away by more environments, but this is an expectation and not evidence.

Why not just raise SPiDR's pessimism weight or widen the randomization until every seed is safe? Turning up pessimism uniformly can buy safety at the cost of "safe but useless" policies. Quanyi Li et al. document this pathology for reward penalties, and the research file flags over-randomization as producing frozen behavior. Uniform pessimism might still be enough, so we trace the full return-versus-violation curve as the pessimism weight and randomization width increase. We then check whether selection at a moderate setting lies above that curve. If it does not, uniform pessimism wins.

Why not just wrap the policy in a safety filter at deployment? Filters such as the Hamilton–Jacobi shield in Sim-to-Lab-to-Real need a reachability model and tuned thresholds. They are complementary to our method, because a filter still benefits from a base policy that rarely triggers it. We report shield intervention rate per seed on the navigation tasks as a secondary metric, which shows whether seed selection reduces reliance on the filter.

Proposed Experiments

We use four task families. The first is Safety-Gymnasium navigation (PointGoal2, which SPiDR already ported to MJX, and CarGoal) and the velocity-constrained locomotion tasks. The second is RRLS-style physics perturbations through the SuReLI/RRLS parameter wrappers on Hopper, Walker2d, and Ant, with a velocity or joint-limit cost added. The third is a quadruped locomotion task modeled on SPiDR's Go1 setup. The fourth is a race-car obstacle task. Each method gets 30 seeds per task.

The first decisive check is the decomposition. If across-seed variance in $p_s$ accounts for less than roughly a fifth of total violation variance under the strongest baseline, seed selection cannot matter much, and we stop the selection arm.

The second check is prediction. Across seeds, we compute the rank correlation and the AUROC for detecting proxy-reality unsafe seeds. We do this separately for beyond-support CVaR, in-support CVaR, mean cost, and return. The proposal's key bet holds only if beyond-support stress clearly beats in-support stress.

The third check is the deployed outcome under matched compute, measured against the baselines above. The outcome is the proxy-reality violation rate and return of the policy each pipeline actually ships, reported with bootstrap intervals over repeated draws of the $K$-seed pool.

The fourth check is calibration. Across tasks we track how often accepted seeds exceed $\delta$ on the stress distribution, which should be at most β by construction. We also track how often they exceed $\delta$ on proxy reality. The size of that gap measures how much an in-simulation certificate can be trusted.

The optional hardware run tests a single falsifiable prediction: the highest-scoring rejected seed should violate more often on the car than the lowest-scoring accepted seed.