Feasibility-Paced Domain Randomization for Zero-Shot Safe Transfer

Robots trained in simulation are usually made robust by domain randomization: every training episode draws new physics, such as friction, mass, motor strength, and latency, so the policy cannot overfit to one simulator. When the policy must also obey hard safety limits, such as never hitting an obstacle or never exceeding a joint torque, the usual recipe pairs wide randomization with a constrained RL solver, an algorithm that maximizes reward while keeping expected "cost" below a budget. Our hypothesis is that this pairing fails in a specific way. Some randomized physics draws are so hard that no policy can stay safe in them. Those draws drive the solver's penalty weight on safety upward, and the policy settles into a "safe but useless" crouch: it barely moves anywhere, including in the easy physics where real robots actually live.

We propose a curriculum that widens the randomization only as fast as the policy stays safe in the physics it currently sees. The curriculum keeps the original wide distribution as its target. It stops short of physics regions that turn out to be infeasible and reports them as a map of where safety was certified in simulation. The question is whether this pacing signal, "how much of the current physics distribution is the policy safely solving," is enough to replace two things: a hand-tuned randomization schedule, and the static pessimism that current zero-shot safe-transfer methods rely on. The comparison targets both conservativeness and real-world safety.

Core Claim

Under hard cost constraints, the width of the randomization distribution is not just a robustness knob. It also determines whether the constrained optimization problem is feasible. When the training distribution includes infeasible physics, one Lagrange multiplier (the adaptive weight that trades reward against cost) is shared across all physics draws, so the infeasible draws inflate it and suppress useful behaviour everywhere else. A curriculum whose expansion is gated by estimated per-physics constraint satisfaction should reach the target distribution with higher task return and no more violations than static randomization. It should also match the best hand-tuned schedule without needing per-task schedule tuning. The new ingredient is narrow but specific. The pacing signal is constraint feasibility over physics parameters, not reward or task success. As a byproduct, the final curriculum distribution marks out the physics region the policy has been checked against.

Relation to Prior Work

The closest work falls into three groups. The first group supplies methods we reuse. SPiDR adds a penalty proportional to disagreement across an ensemble of randomized dynamics models to the safety cost. It plugs that pessimistic cost into standard constrained solvers and reports zero-shot hardware safety. Its randomization ranges are static, and its penalty scale must be tuned to avoid frozen agents. We keep its pessimistic cost unchanged and change only what physics the policy trains on and when.

Self-Paced Deep Reinforcement Learning and its predecessor Self-Paced Contextual Reinforcement Learning move a sampling distribution over task parameters toward a target by trading expected reward against KL divergence to that target. They have no notion of cost, and their parameters are goals rather than physics. We borrow the KL-to-target structure but swap the reward term for a feasibility constraint.

The second group motivates the gap. Adaptive Curriculum Dynamics Randomization widens an actuator-failure range whenever average reward passes a threshold. It reports the surprising result that a hard-to-easy order beats easy-to-hard on Ant-v2, but it handles falling only through reward penalties and was tested only in simulation. Active Domain Randomization learns to propose physics where the policy behaves unlike it does in a reference environment. That is useful for robustness, but under constraints it deliberately seeks the infeasible draws we argue are harmful.

The third group is closest in spirit and constrains the design. UARL already uses a progressive randomization curriculum for safe transfer. Its expansion is driven by critic-ensemble uncertainty measured against a small real-robot dataset, so it is not strictly zero-shot, and it does not pose a constrained optimization problem. Our curriculum uses no real data at all. Safe RL via Curriculum Induction curricula over safety interventions in fixed dynamics, and Provably Safe Sim-to-Real Transfer and Sim-to-Lab-to-Real obtain guarantees by collecting real or lab interaction. These works aim at training-time safety on hardware. We aim at deployment-time safety after purely simulated training, where simulated violations are free but the shape of the training distribution still matters.

Research Program

Let $\xi$ denote the vector of physics parameters, $\mu(\xi)$ the target randomization distribution a practitioner would otherwise use statically, and $p_k(\xi)$ the curriculum distribution at stage $k$. For policy $\pi$, $J_r(\pi,\xi)$ and $J_c(\pi,\xi)$ are its expected return and expected pessimistic cost (the SPiDR cost) under physics $\xi$, and $\epsilon$ is the cost budget. The policy is trained on $p_k$ with PPO under a PID-controlled Lagrange multiplier. Between policy updates, the curriculum solves

$$p_{k+1} = \arg\min_{p} \; \mathrm{KL}(p \,\|\, \mu) \quad \text{s.t.} \quad \Pr_{\xi\sim p}\big[\hat J_c(\pi,\xi) > \epsilon\big] \le \delta, \qquad \mathrm{KL}(p \,\|\, p_k) \le \eta,$$

where $\hat J_c$ is a learned estimate of per-physics cost, $\delta$ is the tolerated fraction of unsafe physics, and $\eta$ is a trust-region step size. In words, the curriculum moves as close to the target as it can while keeping almost all of its mass on physics where the current policy is safe. When the policy can no longer expand, the support of the final $p$ relative to $\mu$ is the feasibility map.

The estimator $\hat J_c$ is a small regression model, either a Gaussian process or an ensemble MLP, from $\xi$ to episode cost. It is fit on rollouts the curriculum already pays for, and those rollouts are charged to its sample budget. This sidesteps the "safety paradox" that starves state-level cost critics of violation data: the frontier of $p_k$ always contains borderline physics, so the estimator keeps seeing near-violations exactly where it needs accuracy.

To handle multimodal feasible sets, which SPDL's Gaussian families cannot represent, we parameterize $p$ as a particle set reweighted toward $\mu$. We will test whether the simpler Gaussian version suffices.

The work proceeds in three steps. The first is a diagnostic study that tests whether the freezing mechanism exists at all. The second builds and ablates the curriculum. The third is a held-out "reality" evaluation in simulation, followed by optional hardware deployment on a 1/10-scale RC car of the kind SPiDR used, if one is available. The hard-to-easy question raised by ACDR becomes one controlled experiment rather than a design choice: we initialize $p_0$ either at nominal physics or at the hardest feasible region found by a short search, and compare.

Baselines

The naive reference is uniform domain randomization over the full target $\mu$, trained with PPO and a PID Lagrangian. This is the setting in which Safety Gymnasium reports that PPO-Lag achieved a 98% reduction in safety violations versus PPO with a 45% drop in total reward on velocity tasks. Every curriculum is run on top of the same solver so that only the training distribution differs.

The strongest existing method is SPiDR with static randomization over $\mu$. On hardware it reports that it satisfied safety constraints 100% of the time, and that standard randomization produced 26.1x more violations on the Unitree Go1 and 6.2x more on the RaceCar (SPiDR). There is no reported result on Lacuna for SPiDR on PointGoal2 in MJX or on the RWRL suite, so we will reproduce it ourselves. We give its penalty scale $\lambda$ the same tuning budget as our method's $\delta$ and $\eta$: five configurations each, chosen on validation physics. Beating SPiDR means matching its held-out violation rate within statistical error while achieving higher return, with return measured specifically on the nominal-and-near-nominal physics where it is most likely to be conservative.

The manual curriculum baseline is a hand-designed linear ramp of randomization width from nominal to $\mu$ over a fixed fraction of training. We sweep five ramp lengths, report both the best and the median, and also run a two-stage "train narrow, then jump to full" schedule. Our method must match the best ramp on each task without per-task tuning and must beat the median ramp. Otherwise its contribution reduces to "pick a good ramp."

Two baselines isolate the key ingredient. The first is the same curriculum machinery paced by reward, in the style of SPDL and ACDR, rather than by feasibility. The second is our curriculum with $\delta=1$, which removes the feasibility constraint. Every method gets the same total environment steps, including the curriculum's evaluation rollouts, the same network sizes, and five seeds per configuration.

Why not just…?

Why not just tune SPiDR's penalty and keep the randomization static? This is the most serious threat. If the frozen-policy failure is mild, a well-tuned static method is simpler and already safe on hardware. The diagnostic study settles it. We compare the multiplier trajectory, mean speed, and return of static SPiDR on $\mu$ against the same method trained only on physics we know to be feasible. If static SPiDR's return on feasible physics is within noise of the feasible-only oracle, the core premise is wrong, and we will report that rather than add a curriculum.

Why not just hand-schedule the randomization width? Peng et al.'s survey argues that manual curricula carry human bias and do not adapt to learning pace. SPDL shows abrupt staging trapping Ant in a stand-still optimum. Neither result was obtained under cost constraints, though, and a linear ramp is a plausible competitor. The best-of-five ramp baseline is designed to beat our method if a good schedule is all that matters. Our claim survives only if no single ramp length is best across tasks, while feasibility pacing tracks the best ramp everywhere.

Why not just use a separate multiplier per physics region, or drop infeasible draws? If shared-multiplier inflation is the mechanism, a context-binned multiplier could fix it without any curriculum, and pruning $\mu$ to its feasible set would do the same if that set were known. We run the binned-multiplier variant directly. Pruning requires the feasibility map that our curriculum produces, so we evaluate it as "static randomization on our final $p$." If either matches our method, the paper's contribution becomes the feasibility estimator and map rather than the pacing, and we will say so.

Why not just penalize failures heavily in the reward? Safety Gymnasium documents that large penalties produce frozen policies and small ones permit violations. Provably Safe Sim-to-Real Transfer reports that an unconstrained baseline violated safety in 68.5% of iterations versus 0% for a constrained one. We include a penalty-only variant on one task for completeness and do not expect it to compete.

Proposed Experiments

The simulated benchmarks are the ones SPiDR used, so its reported setup is the anchor. They are Safety Gym PointGoal2 ported to MJX and the RWRL Cartpole, Walker, and Humanoid tasks with their safety constraints. We add Ant-v2 with ACDR's actuator-failure coefficient as a constrained task, with the cost defined as torso tilt beyond a limit; this tests the hard-to-easy question under constraints. Each task gets a target $\mu$ that is deliberately wide enough to contain physics we verify to be infeasible by training specialist policies on fixed parameters.

"Reality" is simulated by three held-out test sets. The first is drawn from a shifted distribution inside $\mu$. The second is concentrated near nominal physics. The third lies just outside $\mu$'s support, which probes SPiDR's in-distribution assumption.

The metrics are violation rate (the fraction of test episodes over budget), return on each test set, return on the feasible subset of $\mu$, and the multiplier's trajectory. We also measure how well the feasibility map predicts actual test-time violations, scored as area under the ROC curve of $\hat J_c$ against observed cost on held-out physics.

The decisive checks are four. First, does static randomization on $\mu$ underperform feasible-only training? If not, we stop. Second, does feasibility pacing beat reward pacing and the $\delta=1$ ablation at equal violation rate? Third, does it match the best manual ramp across all tasks while beating the median ramp? Fourth, does the feasibility map flag the outside-support test physics as risky before deployment? That last check would give a zero-data deployment gate, the piece missing between UARL's data-dependent gate and SPiDR's lack of one.

Hardware deployment on an RC car repeats SPiDR's protocol, comparing violation counts and lap speed, and is reported only if the simulated checks pass.