Certifying Stability Across the Randomization Distribution, Not Its Worst Case

Robots trained only in simulation are usually made robust by domain randomization: the simulator's physics (masses, friction, motor strength, delays) is resampled every episode, so the policy cannot overfit to one wrong model. The approach is cheap and often works, but nothing tells you before deployment whether the trained policy will stay stable on the real machine. Control theory has a tool for that question, the Lyapunov function, which is a learned or designed "energy" that must shrink at every step so the state keeps moving toward a safe equilibrium. The trouble is that robust Lyapunov certificates are normally required to hold for every physics setting in the uncertainty set. As the randomization range widens, the set of actions that satisfy all settings at once shrinks until it is empty. Randomization wants wide ranges, and worst-case certificates forbid them.

We propose to train a neural policy and a neural Lyapunov function together, inside the randomized simulator, with gradient-based constrained reinforcement learning. The Lyapunov function plays the role of the dense certificate used in stability-shaped control, and the training method supplies the constraint machinery. The key change is how the constraint is posed. The step-by-step energy-decrease condition is imposed as a constraint that must hold on all but a small fraction $\beta$ of sampled physics settings, rather than on all of them or merely on average along trajectories. After training, a held-out batch of sampled physics settings yields a binomial confidence bound on how much of the randomization distribution the certificate covers. The same bound then decides how wide the randomization can be pushed before stability coverage falls below target. That lets the certificate set the randomization range, where today the range is either tuned by hand or set with real-robot data.

Core Claim

A distribution-level, rather than worst-case, stability certificate turns the conflict between wide domain randomization and Lyapunov robustness into a tunable trade-off. It also replaces "train and hope" with a quantitative pre-deployment statement. Concretely, we claim three things relative to the strongest simple alternative, a randomized-physics policy trained with a Lagrangian safety cost. First, at matched task return, our method gives a lower rate of divergent episodes on held-out physics, including physics drawn from a different simulator. Second, at matched coverage target, it keeps more nominal performance than a worst-case robust certificate. Third, its pre-deployment coverage bound predicts the failure rate actually observed on held-out physics. Matched return and matched coverage are both defined in the Baselines and Proposed Experiments sections. The third claim is the one no existing method in this area provides without real-world data.

Where This Sits

The proposal combines three strands that have so far been used separately.

Stability-shaped learning supplies the certificate. Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning adds a hand-designed control Lyapunov function's one-step decrease to the cost. This made real-robot learning fast, but it explores on hardware and needs a hand-crafted function per robot. Lyapunov-Regularized Reinforcement Learning for Power System Transient Stability co-learns a "stability critic" that predicts decrease violations. We reuse both ideas, the dense one-step decrease signal and a learned certificate, but enforce the decrease as a constraint across randomized physics instead of as a penalty on one system. Switched Linear Systems Meet Markov Decision Processes reports that soft reward penalties fail to prevent divergence. That result constrains our design: we use an explicit constraint with an adaptive multiplier, not a fixed penalty weight.

Constrained reinforcement learning supplies the optimizer. Its algorithms, for example First Order Constrained Optimization in Policy Space and the primal-dual methods analyzed by Ding et al., bound an expected cumulative cost. As the file's gap analysis notes, such a bound permits rare catastrophic episodes on particular physics draws. A Lyapunov-based Approach to Safe Reinforcement Learning uses a Lyapunov function to track a remaining cost budget for one fixed transition model. Our Lyapunov function is control-theoretic, an energy toward an equilibrium, and our constraint ranges over a distribution of models.

Three recent papers motivate the gap most directly. LyEvO certifies a neural policy inside a Lyapunov ellipsoid by statistical model checking during evolution-strategy training and transfers it zero-shot to a cartpole and a quadrotor. It needs a linear model to seed the ellipsoid, took over 31 hours on up to 1,000 CPU cores for the quadrotor, produced actuator chatter, and certifies over states for a nominal model, not over a physics distribution. Safe Nonlinear Control Using Robust Neural Lyapunov-Barrier Functions learns a certificate that holds for every parameter in a bounded set and filters actions with a fast quadratic program. That worst-case requirement is exactly what becomes infeasible as the set grows. Safe Domain Randomization via Uncertainty-Aware OOD Detection widens randomization progressively and gates deployment with critic-ensemble disagreement. It needs a real-world target dataset, and its gate is a heuristic rather than a stability statement.

We differ from all three in the same way. The certified object is the fraction of the randomization distribution on which decrease holds, it is obtained with ordinary policy-gradient compute, and no linear prior or real data is needed.

A fourth paper, Transferable Reinforcement Learning via Probabilistic Latent Embeddings, infers the physics online and relaxes risk as uncertainty shrinks. It is complementary, and we treat it as a competing baseline rather than a component.

Research Program

Let $\theta \sim p_w(\theta)$ denote a physics setting drawn from a randomization distribution of width $w$. Let $V_\phi(s) \ge 0$ be a neural Lyapunov function with $V_\phi(s^*)=0$, and let $\pi_\psi$ be the policy. For each sampled $\theta$ we roll out $\pi_\psi$ and record the per-step decrease margin $m_t = V_\phi(s_{t+1}) - V_\phi(s_t) + \alpha V_\phi(s_t)$ on states inside a fixed region of interest $\mathcal{R}=\{s: V_\phi(s)\le c\}$. Here $\alpha \in (0,1)$ is a required decay rate. A physics setting fails if any in-region step has $m_t > 0$. Training solves

$$\max_{\psi,\phi}\; \mathbb{E}_{\theta\sim p_w}\big[J_r(\pi_\psi;\theta)\big] \quad \text{s.t.} \quad \mathbb{E}_{\theta\sim p_w}\Big[\sigma\big(\max_{t:\,s_t\in\mathcal{R}} m_t/\kappa\big)\Big] \le \beta,$$

where $J_r$ is the task return, $\sigma$ is a sigmoid that makes the per-setting failure indicator differentiable, with temperature $\kappa$, and $\beta$ is the target failure fraction. We optimize with a first-order constrained policy update in the style of FOCOPS and a PID-damped Lagrange multiplier, from the SafePO library in Safety Gymnasium, to limit the multiplier oscillation the file documents for primal-dual methods. $V_\phi$ is trained on the same rollouts with a hinge loss on $m_t$ plus a positivity term. Because $V_\phi$ is not given by hand, no linear model is needed. A small action-smoothness penalty addresses the chatter reported for LyEvO, following the observation in Action Manifold Smoothing that action parameterization amplifies instability.

Certification is separate from training. After training, we draw $N$ fresh physics settings from $p_w$ and $M$ initial states per setting in $\mathcal{R}$, then count the $k$ settings with any decrease violation. The Clopper–Pearson upper bound $\bar\beta(k,N,\delta)$ then states, with confidence $1-\delta$, that the probability a random physics setting from $p_w$ violates decrease is at most $\bar\beta$. The bound comes from treating each setting as an independent Bernoulli trial.

We are explicit about two limits. The check is statistical over sampled states, not a formal proof over the continuum, which mirrors LyEvO's model checking. And it says nothing about a real robot whose parameters lie outside $p_w$.

Randomization width then becomes an output. Starting from a narrow $w$, we widen it, warm-starting training, for as long as $\bar\beta \le \beta$. We report the widest certified width alongside return.

Baselines

The naive reference point is plain domain randomization with PPO at a hand-chosen width, as in Sim-to-Real: Learning Agile Locomotion For Quadruped Robots, with no safety signal. It tells us how often unconstrained randomization already stays stable.

The strongest competitor is PPO-Lagrangian trained under the same randomization. Its cost signal is a hand-written failure indicator: tilt or state bound exceeded, termination, or a velocity limit in the velocity tasks. It is the method a practitioner would actually try, it is implemented in SafePO, and it attacks the same failure mode with less machinery. Safety Gymnasium reports that PPO-Lagrangian achieved a 98% reduction in safety violations with a 45% drop in total reward relative to PPO on its velocity-constrained tasks. On randomized dynamics in RRLS, the file has no reported numerical scores: no reported result found on Lacuna for PPO-Lagrangian or domain randomization scores on the RRLS tasks. We will therefore establish these numbers ourselves.

To be fair, every method gets the same randomization distribution, the same environment-step budget (5 million steps, matching RRLS's protocol), 10 seeds, the same network sizes, and an equal hyperparameter search budget. The cost threshold for PPO-Lagrangian is swept so that we compare at matched nominal return. Beating it means a lower held-out divergence rate at matched return, or higher return at matched divergence rate, outside seed confidence intervals.

The worst-case certificate baseline is a robust neural Lyapunov-barrier controller in the style of Dawson et al., trained over the same parameter box. It tests whether our quantile relaxation actually buys performance and feasibility as width grows.

The context-adaptive baseline is the latent-embedding, risk-adaptive method of Han and Feng. On PointGoal2 under out-of-distribution parameter shifts, that paper reports a cost of 8.86 against a limit of 10, versus about 29.6 for a nominal policy. We reproduce that setting as a sanity check before using the method on our tasks.

Where a cartpole setup is available, LyEvO is the reference for certified sim-to-real stability. It reports 0% simulated violation probability against deep RL baselines with violation rates up to 54%, and a 6.14× larger verified region than linear methods on the Quanser cartpole. We compare region size and compute rather than claiming to beat a 0% violation rate.

Two ablations isolate the key ingredient. The first replaces the distribution-level decrease constraint with the same decrease term used as a fixed-weight cost, which is the Westenbroek-style shaping under randomization. The second keeps the constraint but averages margins over steps instead of taking the per-setting maximum. That turns it back into a cumulative, CMDP-style cost and tests whether the pointwise structure matters.

Why not just…?

Why not just run PPO-Lagrangian under domain randomization with a crash cost? This is the real threat. On low-dimensional tasks a well-tuned crash cost may match us. The expected-cost formulation does allow rare per-setting divergence that averages out, and the file attributes failure rates up to 54% to safe deep RL baselines in LyEvO. But that comparison was not under matched tuning. Our matched-return comparison on held-out physics settles it directly. If PPO-Lagrangian's tail failure rate is statistically indistinguishable from ours, the remaining contribution is the coverage bound alone, and we will say so.

Why not just add a control Lyapunov term to the reward, as in Westenbroek et al.? A fixed weight reproduces the fragile balance Li et al. describe: too small and rare instability is accepted, too large and the agent freezes. Wu et al. find soft penalties insufficient against divergence. The fixed-weight ablation above measures exactly this, with the weight swept so the comparison is fair.

Why not just train ordinary domain randomization and certify afterwards with the same binomial test? Post-hoc testing gives a bound, but with no learned decrease target in training it can only certify a hand-chosen safe set, not a Lyapunov region. It also gives no lever to improve coverage except retraining blindly. We run this version too, using the same held-out test with a separately fitted $V_\phi$ on the frozen policy. If it achieves similar coverage at similar width, in-the-loop training is not needed and the proposal shrinks to a verification protocol.

Why not just infer the physics online and adapt? Online inference as in Han and Feng can beat any fixed policy on nominal performance. It assumes parameters are fixed within an episode, though, and its guarantee is on expected excess cost, not on state boundedness. We test both methods on mid-episode parameter jumps, and we expect the methods to compose, since our certificate could condition on the inferred context.

Proposed Experiments

The main suite uses RRLS (paper, code), whose parameter-modifying environment interface covers Inverted Pendulum, Hopper, Walker2d and Ant with 1D to 3D uncertainty sets. We add the velocity-constrained tasks of Safety Gymnasium under the same physics randomization, and PointGoal2 for the comparison with Han and Feng. As a stand-in for reality, every policy trained in MuJoCo is also evaluated on held-out parameter settings outside the training box and in a second simulator, following LyEvO's practice of using two simulators. Where hardware is available, the Quanser cartpole provides a real-robot check against LyEvO's published numbers.

Metrics are nominal return, return averaged over held-out physics settings, the fraction of held-out settings with any divergent episode, the certified coverage bound $\bar\beta$, the widest certified randomization width, action total variation as a chatter measure, and wall-clock training compute.

Four checks decide the outcome. First, calibration: across tasks and widths, the observed violation fraction on held-out physics must fall below $\bar\beta$ at roughly the nominal confidence. If it does not, the certificate is not trustworthy and the claim fails. Second, the width–feasibility curve: as width grows, the worst-case certificate baseline should become infeasible or lose return sharply, while our method degrades gracefully at fixed $\beta$. Third, the matched-return comparison against PPO-Lagrangian on held-out physics tails decides whether the stability constraint does anything a crash cost does not. Fourth, the post-hoc-certification ablation decides whether training with the certificate matters, or only testing with it.