When a robot controller is trained in simulation and deployed on hardware with no real-world practice, which is called zero-shot sim-to-real transfer, the standard trick is domain randomization. The simulator's physics (friction, mass, motor delay) is varied during training so the policy learns to cope with whatever reality turns out to be. That trick targets average performance, not safety. The one method in our file that does target safety, SPiDR, adds a penalty wherever an ensemble of randomized simulators disagrees about what happens next. Its safety margin comes from a hand-tuned coefficient, and it assumes the real robot's physics lies inside the randomized range. We propose a margin that has a physical meaning. We learn a safety critic, a network predicting future constraint cost, that is conditioned on the physics parameters and is forced to change slowly as those parameters change. Its worst case over the whole parameter range, and some distance beyond it, can then be bounded from a finite set of simulated samples plus a slope-times-distance term.
The central observation links this to policy smoothness. How fast a closed-loop system's safety cost changes with physics depends on how sharply the controller reacts to state changes. A jittery policy amplifies small physics differences into large trajectory differences. So bounding the policy's Lipschitz constant, meaning the most its output can change per unit change of input, does more than stop actuator chatter. It is what keeps the safety margin small enough to be useful. The claim we test is that smoothness and certifiable robustness are the same resource, and that spending it deliberately gives safer zero-shot transfer at less cost in task performance than ensemble-disagreement pessimism.
Core Claim
Let $\xi$ be the vector of simulator parameters and $\Xi$ the randomization range. Let $Q_c^\pi(s,a,\xi)$ be the expected discounted constraint cost of policy $\pi$ from state $s$ and action $a$ under physics $\xi$. This critic sees $\xi$ during training only. The actor never does, as in standard asymmetric actor-critic setups. If $Q_c^\pi$ is $L_\xi$-Lipschitz in $\xi$ and we evaluate it at samples $\xi_1,\dots,\xi_K$ that cover $\Xi$ enlarged by a margin $\delta$ to within radius $r$, then
$$\bar Q_c(s,a) \;=\; \max_{k} Q_c^\pi(s,a,\xi_k) \;+\; L_\xi\, r \;\ge\; \sup_{\xi \in \Xi_\delta} Q_c^\pi(s,a,\xi),$$
where $\Xi_\delta$ is the set of parameters within distance $\delta$ of $\Xi$. We use $\bar Q_c$ as the constraint in a standard constrained Markov decision process (CMDP) solver, the framework that maximizes reward subject to a budget $\epsilon$ on expected cost.
The link to the actor comes from a standard perturbation argument over a finite horizon $H$. Trajectory divergence under two physics settings grows roughly like $(L_{P,s} + L_{P,a} L_\pi)^t$ per step. Here $L_{P,s}$ and $L_{P,a}$ are the sensitivities of the dynamics to state and action, and $L_\pi$ is the policy's Lipschitz constant. Driving $L_\pi$ down therefore shrinks the achievable $L_\xi$, which shrinks the margin term, which lets the constrained solver be less conservative. This replaces SPiDR's unitless $\lambda$ with two interpretable quantities: a slope we enforce and a coverage radius we choose. The $\delta$ term also gives an explicit, if heuristic, allowance for real physics slightly outside the simulated range, which SPiDR does not attempt.
We are careful about what is claimed. The bound is exact only if $L_\xi$ is a true bound. We enforce it with a gradient penalty and check it empirically. We do not use spectral-norm products, which are loose. The result is an empirical certificate, not a formal one. We also avoid the contraction requirement $\gamma L_P < 1$ that limits Policy gradient in Lipschitz Markov Decision Processes, which proves that smooth dynamics make returns smooth. We learn $L_\xi$ directly on finite-horizon cost values rather than deriving it from the dynamics.
Relation to Prior Work
Three papers motivate the gap. SPiDR augments cost with ensemble next-state variance and reports 100% real-world constraint satisfaction on a Unitree Go1 and a 1/10 scale RaceCar. Standard domain randomization had 26.1x and 6.2x more violations on those platforms. SPiDR is our main competitor. It has no smoothness control, its $\lambda$ is tuned by hand, and it assumes in-support reality. LyEvO certifies safe regions for sim-to-real quadrotor control with evolutionary search and statistical model checking. It explicitly concedes that its neural controllers jitter, and it needed 31 hours on up to 1,000 CPU cores. That is evidence that certification without smoothness is both expensive and hard on hardware. Provably Safe Sim-to-Real Transfer gives zero-violation guarantees but only in a tabular 5x5 gridworld, under a separable-shift assumption that fails for continuous physics.
Two papers supply the method. Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies (LCP) penalizes the actor's input gradient norm and cut jitter from about 42 to about 3.2 on several humanoids with one coefficient. It has no constraints or safety budget. Regularizing Action Policies for Smooth Control with Reinforcement Learning (CAPS) achieves smoothness with perturbation-based spatial and temporal losses. We reuse LCP's penalty on the actor and the same penalty form on the critic's $\xi$-gradient. Neither paper connects smoothness to safety margins.
One paper constrains the design. Statistical Guarantees for Offline Domain Randomization shows that fitting randomization ranges from real logs is consistent only under uniform Lipschitz continuity of the dynamics in $\xi$. That supports the idea that Lipschitz structure in parameter space is the right handle. It also warns that misspecified simulators converge only to the nearest projection, which is exactly the case our $\delta$ margin targets. Breaking Safety Paradox with Feasible Dual Policy Iteration notes that safer policies starve safety critics of violation data. A $\xi$-smooth critic partly answers this, because boundaries learned at benign $\xi$ constrain the critic at harsh $\xi$ through the slope bound. It also poses a risk: the critic may be smooth but wrong where there is no data. We measure that directly.
Research Program
We would build four components. The first is a parameter-conditioned cost critic $Q_c(s,a,\xi)$ trained on domain-randomized rollouts, with a hinge penalty $\max(0,\|\nabla_\xi Q_c\|^2 - L_\xi^2)$ in the style of LCP. The second is an actor under LCP's state-gradient penalty with bound $L_\pi$. The third is a covering scheme: a fixed low-discrepancy set of $K$ parameter samples over $\Xi_\delta$, with $r$ computed from it. The fourth plugs $\bar Q_c$ into PPO-Lagrangian with the PID-damped multiplier used in Safety Gymnasium, so that multiplier oscillation is not confused with our effect. We use the same solver backbone SPiDR supports, so the only change is the pessimism term.
The analysis sweeps $L_\pi$ and measures three things: the smallest $L_\xi$ the critic can meet without its fitting error rising, the resulting margin $L_\xi r$, and the reward at a fixed safety budget. The prediction is a U-shape. Too little actor smoothness makes the margin large and the policy freezes. Too much makes the policy unable to perform agile recoveries, the expressivity trade-off LCP's authors flag. The interior optimum should beat both SPiDR and SPiDR plus smoothing.
Baselines
The naive reference is plain domain-randomized PPO-Lagrangian: the same randomization range, the same constraint budget, and no pessimism or smoothness term. It shows how much any of the machinery matters. For comparison, the Safety Gymnasium authors report PPO-Lag reducing violations by 98% relative to unconstrained PPO at a 45% reward cost in velocity tasks.
The strongest existing method is SPiDR on the same backbone, with its ensemble of at least 8 randomized simulators. The published hardware result is 100% constraint satisfaction on Go1 and RaceCar. On the simulated PointGoal2 port we found no reported result on Lacuna, so we run it ourselves. Each method gets the same simulator-step budget and the same number of hyperparameter trials. SPiDR's trials sweep $\lambda$; ours sweep $L_\pi$, $L_\xi$ and $\delta$. Selection uses in-support validation physics only. Beating SPiDR means matching its violation rate on in-support held-out physics while achieving higher return. Alternatively, it means a strictly lower violation rate on out-of-support physics at equal return.
The key ablation is SPiDR plus the LCP actor penalty with the same $L_\pi$, which isolates whether our critic-side Lipschitz margin adds anything beyond smoothing the actor. A second ablation drops the $L_\xi r$ term and keeps only the max over samples, which isolates the Lipschitz extrapolation. A third ablation uses our margin with an unsmoothed actor, which tests the central claim that actor smoothness is what makes the margin affordable. For single-environment constrained optimization, FOCOPS is a reference solver. It reports 1106.1 reward on Humanoid-Circle within a cost limit of 50, and we can substitute it as the backbone to check that results are not tied to PPO-Lagrangian.
Why not just…?
Why not just add LCP's gradient penalty to SPiDR? This is the obvious combination and a real threat. If SPiDR plus LCP matches our method on both in-support and out-of-support violations at equal return, then our contribution reduces to an interpretable parameterization of pessimism, not a better one. The ablation above settles this directly. Our specific prediction is that the two methods tie in-support but diverge out-of-support. Ensemble variance is zero-information outside the ensemble's range, while the slope-times-distance term grows with $\delta$.
Why not just take the worst case over many sampled physics settings? Worst-case or tail-risk domain randomization without a slope bound is simpler, and with enough samples it may approach our margin inside $\Xi$. It gives no statement about points between samples or beyond the range, and its cost grows with $K$. The max-only ablation, evaluated on a dense held-out grid and on out-of-support physics, shows whether the $L_\xi r$ term buys safety that more samples cannot.
Why not just filter the actions or penalize action differences in the reward? CAPS and LCP both report that low-pass filters add phase lag and dampen exploration, and that reward-level smoothness penalties need robot-specific tuning. More importantly, a filter smooths the output after the fact without changing the critic's sensitivity to physics, so it cannot shrink the certified margin. We include a filtered PPO-Lagrangian run to confirm this on jitter score and violations.
Proposed Experiments
The main simulated testbeds are Safety Gym PointGoal2 in MJX and the Real-World RL tasks, both used by SPiDR, plus Safety-Gymnasium velocity-constrained locomotion. Physics parameters (friction, mass, actuator delay) are randomized over $\Xi$. We evaluate on three held-out sets: a dense grid inside $\Xi$, a shell at distance up to $\delta$ outside it, and a farther shell beyond $\delta$ that no method should claim to handle. Robust-Gymnasium's action and observation disruptions give a second, independent source of shift. If a Go1 is available, we repeat SPiDR's hardware protocol and add latency and payload changes outside the training range.
We report four metrics. The first is constraint violation rate per episode. The second is return at the fixed budget. The third is a jitter score computed as in LCP. The fourth is margin calibration: the gap between predicted $\bar Q_c$ and the realized worst-case cost on the held-out grids.
Three results would be decisive. First, if realized worst-case cost exceeds $\bar Q_c$ on the in-range and $\delta$-shell grids more than rarely, the enforced $L_\xi$ is not a real bound and the certificate claim fails. Second, if the $L_\pi$ sweep shows no interior optimum, or the unsmoothed-actor ablation reaches the same margin and return, the claim that smoothness pays for certifiability is false. Third, if SPiDR plus LCP ties us on the out-of-support shell, the Lipschitz extrapolation is not pulling its weight. In that case we would report the simpler combination as the recommended method.
References
- Policy gradient in Lipschitz Markov Decision Processes
- SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfer
- LyEvO: Lyapunov-Guided Evolutionary Optimization for Safe and Robust Sim-to-Real Policy Learning
- Provably Safe Sim-to-Real Transfer
- Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies
- Regularizing Action Policies for Smooth Control with Reinforcement Learning
- Statistical Guarantees for Offline Domain Randomization
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space