When a reinforcement learning agent learns on a physical system, every intermediate policy is actually executed. Optimization theory usually asks whether the final policy is good. For a quadrotor or a legged robot learning online, the question that matters is whether each policy along the way keeps the machine stable. A common way to impose stability is a Lagrange multiplier: a learned penalty weight that rises when a constraint is violated and falls when it is satisfied. These multipliers are known to "yo-yo": the policy alternates between aggressive, constraint-breaking behavior and over-cautious retreat. In an abstract constrained problem this oscillation is a convergence nuisance. On hardware it means that some intermediate policies destabilize the system.
We propose to enforce a stability condition during online training with a multiplier that has two properties. It is a function of the state rather than a single number, and its update is designed so that the current iterate itself converges rather than merely its running average. The stability condition is a Lyapunov decrease requirement. A Lyapunov function is an "energy" that must shrink along trajectories for the system to be stable. We require it to decrease on average over a short window of $M$ steps rather than at every step, which allows the temporary energy increases that swing-up and obstacle avoidance need. The claim to test is this: once stability is written as a state-wise constraint, oscillation of the learning dynamics and instability of the robot become the same failure. A state-dependent, dampened, optimistic multiplier can therefore cut the stability violations of every training iterate, not only the final one, without giving up return.
Core Claim and What Is New
Three lines of work each solve one piece of this problem.
The first gives last-iterate convergence for constrained problems whose constraint is a single cumulative cost. ReLOAD applies optimistic mirror descent, using the extrapolated gradient $2\nabla L^k - \nabla L^{k-1}$, to both policy and multiplier. It shows that optimism must be symmetric to suppress yo-yo dynamics in deep RL. C-PG adds a ridge penalty $-\frac{\omega}{2}\|\lambda\|^2$ to the dual and proves dimension-free last-iterate global convergence. RPG-PD/OPG-PD prove sublinear and linear last-iterate rates, and use a safety buffer to reach zero violations after finitely many iterations. All three use one scalar multiplier on one discounted cost, and the theory assumes a generative model (the ability to sample any state on demand).
The second line enforces stability through Lyapunov functions. LBAC learns a combined Lyapunov-barrier function and balances it against reward with a plain Lagrangian. Its authors report early-training stability violations and give no convergence guarantee for the multiplier loop. Lyapunov cost shaping folds the one-step Lyapunov difference into the cost with no multiplier at all. The Lyapunov-based safe RL approach imposes strict per-step state-wise budgets derived from a safe baseline policy. It is restricted to discrete actions, and it becomes conservative when the task needs non-monotone trajectories.
The third line relaxes the decrease requirement. Generalized Lyapunov functions replace one-step decrease with an $M$-step weighted-average condition, but only to certify policies after training.
Our proposal differs from the first line in the object being constrained. We constrain a state-wise dynamical invariant, not a cumulative cost, so the multiplier becomes a function $\lambda_\phi(s)$. It differs from the second line in using regularized, optimistic dual dynamics with a last-iterate target in place of plain Lagrangian ascent. It differs from the third line by turning the multi-step condition into a training-time exploration constraint.
Concretely, $V$ is a candidate Lyapunov function, either built from an approximate model as in Lyapunov Design or co-learned. For each visited state $s$ we define the multi-step decrease violation
$$g_\pi(s) = \mathbb{E}_\pi\Big[\textstyle\sum_{j=1}^{M} w_j V(s_{t+j})\Big] - V(s) + \alpha(s),$$
where $w_j \ge 0$ are fixed averaging weights summing to one and $\alpha(s) > 0$ is a required decrease margin. The training objective is the saddle problem
$$\max_\pi \min_{\lambda_\phi \ge 0} \; J_r(\pi) - \mathbb{E}_{s \sim d_\pi}\big[\lambda_\phi(s)\, g_\pi(s)\big] + \frac{\omega}{2}\mathbb{E}_{s\sim d_\pi}\big[\lambda_\phi(s)^2\big].$$
In this objective, $J_r$ is the task return, $d_\pi$ is the state-visitation distribution of the current policy, and $\omega > 0$ is the ridge strength borrowed from C-PG. Both players take optimistic steps, as ReLOAD requires.
Research Program
The theoretical piece targets the simplest case where a clean statement is plausible: tabular states with a generative model and $M=1$. With $M=1$ the one-step decrease constraint is linear in the policy at each state. Weighted by visitation, it is linear in the occupancy measure, so the regularized problem inherits the convex-concave structure used by OPG-PD. We aim to extend the linear last-iterate rate of OPG-PD to $|S|$ state-wise constraints. We also aim to show that the resulting bias from $\omega$ can be absorbed into a margin on $\alpha$, playing the role of that paper's safety buffer. The case $M>1$ makes the constraint nonlinear in the policy, and there we will state only empirical claims.
The algorithmic piece is a continuous actor-critic built on ReLOAD's MPO/MDPO backbone, with three additions. The first is a multiplier network $\lambda_\phi$ trained on replay states. The second is a critic for $g_\pi$ that bootstraps the $M$-step expected energy. The third is the retained previous-iteration copies of both the policy-side and multiplier-side gradients needed for the optimistic extrapolation.
The evaluation piece records the stability of every iterate. Measuring only the last checkpoint would miss exactly the phenomenon this proposal is about.
Baselines
The naive reference points are unconstrained PPO and PPO-Lagrangian. For PPO-Lagrangian, the cost is the indicator of Lyapunov-decrease violation and the multiplier is a single scalar. On Safety Gymnasium velocity tasks, PPO-Lag is reported to achieve "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward" (Safety Gymnasium). This gives a published anchor for how much a naive Lagrangian costs in return.
The strongest competitor on the oscillation question is ReLOAD run on that same scalar Lyapunov-violation cost. It isolates whether the state-wise multiplier and the $M$-step constraint add anything beyond optimism alone. ReLOAD is reported to be "the only method to achieve high reward while strictly obeying the height limit" on Humanoid (Height). No numeric return or cost figure for ReLOAD was found on Lacuna, so we will reproduce it on its own oscillating control suite before comparing. C-PG, which uses dual ridge regularization without optimism, is run in the same way.
The strongest competitor on the stability question is LBAC. It converged in "approximately 2,300 episodes" on the 2D quadrotor task, where RCPO "failed to converge within 3,000 episodes" (LBAC). Next is Lyapunov cost shaping, which reached target performance in "2,000 steps" versus "24,000 steps" for standard RL (Lyapunov Design). The last is CPPO-PID, which Safety Gymnasium shows reduces oscillation relative to naive Lagrangians.
All methods share the same backbone network sizes, the same candidate $V$, the same environment-step budget, and the same number of hyperparameter trials, with ten seeds each. CPPO-PID receives a tuning grid over its three gains of the same size as ours over $\omega$ and step sizes.
Two ablations isolate the key ingredients. One uses a state-wise multiplier without optimism or ridge, to test whether state-dependent multipliers oscillate worse on their own. The other varies $M$ between 1 and 3 with the full method.
Beating the field means achieving three things at once. First, a lower worst-iterate and cumulative training-time violation rate than ReLOAD and LBAC, with non-overlapping seed confidence intervals. Second, final return within noise of the best baseline. Third, a certified region of attraction for the final iterate at least as large as that of the best baseline.
Why not just…?
Why not just plug the Lyapunov violation into ReLOAD as an ordinary cost? This is the most serious threat, and it is the main head-to-head comparison. The concern with a scalar dual is that a discounted average of violations can stay small while violations concentrate in the few states exploration pushes toward the edge of the stable region. A single multiplier can only raise the penalty everywhere, which makes the policy timid elsewhere. If scalar ReLOAD matches our per-iterate violation profile on the quadrotor and Humanoid tasks, the state-wise multiplier is unnecessary. The contribution would then shrink to "use ReLOAD for stability costs," and we would report it that way.
Why not just shape the cost with the Lyapunov difference and skip multipliers? Lyapunov shaping is fast and works on hardware. However, it fixes the stability-reward trade-off in advance and enforces one-step decrease. The fixed trade-off is exactly the scalarization that OPG-PD shows cannot represent stochastic constrained optima. One-step decrease is the condition that the generalized Lyapunov paper shows swing-up tasks violate, with its certified region of attraction on the pendulum growing from 42.9 to 89.2 when moving to $M=3$. On pure stabilization tasks, shaping may well tie us, and we expect to say so. The decisive test is on tasks that combine stabilization with reward that conflicts with stability, such as quadrotor navigation around obstacles.
Why not just use a PID-controlled multiplier? CPPO-PID damps oscillation in practice (Safety Gymnasium). It offers no last-iterate guarantee, however, and it adds three gains that must be retuned per task. The equal-budget tuning comparison will show whether it matches us. If it does, the remaining argument is the tabular theory, and we would not overclaim beyond that.
Proposed Experiments
The primary testbeds are the generalized-Lyapunov benchmarks (inverted pendulum, cartpole, 2D quadrotor) from the certification paper and the 2D quadrotor obstacle task from LBAC. These are chosen because they allow formal certification of the final policy with α,β-CROWN, in addition to the empirical measures below. To test transfer beyond control benchmarks, we use ReLOAD's oscillating control suite (Walker, Quadruped, Humanoid with height and velocity constraints) and the Safety Gymnasium velocity tasks. The RRLS perturbed environments (SuReLI/RRLS) probe whether the benefits survive model mismatch in $V$.
The metrics are the following. The fraction of training episodes, across all iterates, in which the $M$-step decrease is violated inside the target region. The number of divergence events. The worst single-iterate violation rate. The oscillation amplitude of the average multiplier. Final return. The certified region-of-attraction volume of the final policy.
Two checks decide the outcome. The first is the scalar-versus-state-wise comparison under ReLOAD on the quadrotor obstacle task. The second is the $M=1$ versus $M=3$ ablation on pendulum swing-up, where we predict that $M=1$ either stalls the swing-up or forces the multiplier to oscillate. If neither effect appears, the proposal's central mechanism is not doing the work, and the result is still informative as a negative finding about state-wise duals.
References
- ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPs
- Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning
- Last-Iterate Convergent Policy Gradient Primal-Dual Methods for Constrained MDPs
- Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions
- Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning
- A Lyapunov-based Approach to Safe Reinforcement Learning
- Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark