In safe reinforcement learning, an agent must maximize reward while keeping an expected "cost", such as collisions or speed-limit breaches, below a fixed budget. The constraint has to hold during training, not only once training ends. A badly chosen learning rate does more harm here than in ordinary deep learning. A step that is too large does not just make the loss spike; it moves the policy into behaviour that breaks the constraint on real episodes. Tuning the learning rate by grid search is itself unsafe, because every failed configuration racks up violations. Separately, the optimization literature has built step-size rules that correct themselves even when the starting learning rate is off by several orders of magnitude. Polyak-type rules are one family: they set the step from the distance between the current loss and a known target value. Their weak point is that this target, the optimal loss $f^*$, is usually unknown.
This note proposes that safe exploration removes that weakness. The cost budget $b$ is a known target, given by the problem. We propose a step-size rule that wraps any base optimizer. When the policy is feasible, the rule takes the largest step that a locally estimated quadratic model of the cost predicts will stay under the budget, after subtracting a margin for noise. When the policy is infeasible, the same model gives a Polyak step aimed at the budget, which drives the policy back to feasibility. The curvature in the model is estimated online from consecutive iterates, Barzilai–Borwein style, so no smoothness constant needs tuning. One rule should therefore deliver two kinds of initialization robustness that current safe-exploration methods lack: insensitivity to the base learning rate, and recovery from an infeasible starting policy.
Core Claim
Our central claim is that a budget-targeted, curvature-estimated step-size cap makes constrained policy optimization on standard safe-RL benchmarks robust across at least three orders of magnitude of base learning rate. Robust here means that cumulative training-time cost stays near the budget and final return stays near the best tuned baseline. The same rule should recover from deliberately infeasible initial policies with bounded cost accumulated during recovery.
The closest existing work falls into two groups. The first adapts step sizes in constrained RL, but none sets the step from the budget with online-estimated curvature while also handling infeasible starts. Adaptive Primal-Dual Method for Safe Reinforcement Learning sets the policy learning rate as a fixed algebraic function of the penalty weight on cost (the Lagrange multiplier), scaling it as $1/\lambda$ or $1/\lambda^2$. It never looks at the cost value, the gradient, or the curvature, and it does not prevent violations during training. A Safe Exploration Approach to Constrained Markov Decision Processes (LB-SGD) shrinks the step near the boundary of a log-barrier objective, which is the method most like ours. It assumes known smoothness constants and a strictly feasible initial policy, still needs a tuned base schedule, and its guarantees cover settings close to tabular or softmax policies.
The second group supplies the method. From MoMo we reuse the idea of truncating a step when a momentum-averaged linear model predicts overshooting a known bound. From SANIA we reuse Polyak steps of the form (value − target)/‖gradient‖². From Barzilai-Borwein Step Size for Stochastic Gradient Descent we reuse secant curvature estimates. In each of these the bound is unknown or only estimated; in the constrained setting it is given exactly.
Research Program
The mechanism. Let $d_t$ be the update direction proposed by the base optimizer, such as an Adam step on the PPO or Lagrangian surrogate. Let $\hat C_t$ be the estimated cost return of the current policy, and $\bar g_t$ a momentum average of the cost-return gradient. Let $\hat\sigma_t$ be the standard error of $\hat C_t$ from the current batch, $\kappa$ a confidence multiplier, and $\hat L_t$ a curvature estimate of the cost along recent steps. We compute $\hat L_t$ as the absolute secant ratio $|\langle g_t - g_{t-1}, \theta_t-\theta_{t-1}\rangle|/\|\theta_t-\theta_{t-1}\|^2$, smoothed as in SGD-BB. When the policy is predicted feasible, the step multiplier is
$$s_t=\max\Big\{s\ge 0:\ \hat C_t + s\,\langle \bar g_t, d_t\rangle + \tfrac{\hat L_t}{2}\,s^2\|d_t\|^2 \le b-\kappa\hat\sigma_t\Big\},$$
which has a closed form as the positive root of a quadratic. The parameter update is $\theta_{t+1}=\theta_t+\min(s_t, s^{R}_t)\,d_t$. Here $s^{R}_t$ is the reward-side step: MoMo's truncated step on the reward surrogate, using MoMo's online estimate of the unknown reward bound. The cost-side cap $s_t$ binds near the boundary, and the reward-side step $s^R_t$ sets the scale far from it.
When $\hat C_t > b-\kappa\hat\sigma_t$, the rule switches to a recovery step. This step moves along $-\bar g_t$ by the Polyak amount $(\hat C_t-b+\kappa\hat\sigma_t)/(\|\bar g_t\|^2+\hat L_t)$, the step that the quadratic model predicts will land at the margin-adjusted budget. This mirrors the restoration phase of interior-point solvers. The margin $\kappa\hat\sigma_t$ is how we take the safety paradox seriously. Breaking Safety Paradox with Feasible Dual Policy Iteration shows that safer policies see fewer violations, so cost estimates get noisier. The margin grows exactly when the cost model deserves less trust. The rule needs only the budget and $\kappa$, which is a confidence level rather than a scale.
Analysis. Like MoMo and SVRG-BB, we will not claim convergence results for non-convex deep policies. The theoretical target is narrower and checkable. Suppose the secant estimate upper-bounds the true local curvature along the step, and the cost estimate's error is sub-Gaussian. Then each feasible step keeps the true cost under budget with probability $1-\delta(\kappa)$, which gives a union bound over $T$ steps of the same $T\delta$ form as LB-SGD's guarantee. Under the same conditions, recovery reaches feasibility within a number of steps logarithmic in the initial excess cost. The empirical work then measures how often the curvature condition fails in practice. We will log the realized cost after each step against the model's prediction. This is the assumption most likely to break, because curvature measured in parameter space can understate how sharply the cost changes along a new direction.
Infeasible starts. Following the gap named in the research file, we will build infeasible initial policies in three ways: pretraining unconstrained PPO, scaling up the action noise, and initializing toward hazards. We then measure recovery. Existing barrier and state-wise methods (IPO, Backward Value Functions, LB-SGD) assume a feasible start and have no defined behaviour without one.
Baselines
The naive reference point is the Lagrangian PPO baseline with a fixed learning rate. Safety-Gymnasium reports that on velocity tasks this baseline gives "a 98% reduction in safety violations compared to standard PPO, with only a 45% drop in total reward."
The PID-stabilized variant, CPPO-PID, damps oscillations in the penalty weight. It is the practical default, so we will also wrap it with our rule.
The strongest competitor on learning-rate sensitivity is the PAPD algorithm with its InvLin and InvQua schedules from Adaptive Primal-Dual Method. It is the only published method that adapts the policy learning rate inside constrained RL. It reports that InvQua is best in return and cost on SafetyCarRun-v0, but the file contains no numbers for it, so we will rerun it on its own four Bullet-Safety-Gym environments. On safety during training, the strongest competitor is LB-SGD. We compare against it on its 6x6 gridworld, where it guarantees violation probability below $mT\delta$ with $\tilde{\mathcal O}(\varepsilon^{-6})$ samples, and on Circle2D. For final return under a cost limit, FOCOPS is the reference, at 1106.1 reward on Humanoid-Circle with cost limit 50, against 671.0 for CPO and 697.5 for TRPO-Lagrangian (FOCOPS).
Fairness rules apply to all methods. Every method gets the same environment steps, network architecture, and seeds (at least ten). Every method is evaluated across the same base learning-rate grid, from $10^{-6}$ to $10^{-1}$ in half-decade steps. Baselines get their own extra hyperparameters tuned at the best learning rate with an equal tuning budget.
Two ablations isolate the key ingredient. The first wraps PPO-Lag with MoMo alone, giving reward-side robustness with no cost cap. The second uses our rule with $\hat L_t=0$, a purely linear cost model, to test whether the secant curvature term matters. Beating the baselines means two things. Our method must keep a learning-rate band at least ten times wider than the best baseline's. In that band, final return must be within 10% of the baseline's best tuned return, and cumulative training cost must be no higher. Matching tuned performance at a single learning rate is not enough.
Why not just…?
Why not just grid-search the learning rate? At the best grid point a tuned PPO-Lag may well match us on final return; we expect it to. The sweep, however, is not free in this setting. Every diverging configuration is a run of unsafe exploration. We will report the total cost accumulated across the entire sweep next to the cost of a single run of our method. A second problem is that the right policy learning rate drifts as the penalty weight grows, which is the core finding of Adaptive Primal-Dual Method, so a fixed best value is a compromise.
Why not just use CPO's trust region with its recovery step? This is the most serious threat, and we say so plainly. As described in the FOCOPS comparisons, CPO also linearizes the cost and has a recovery step for infeasible iterates. The difference is how the step size is chosen. CPO's step magnitude comes from a trust-region radius $\delta$ that must be tuned, it uses a single-batch linear model with no noise margin, and it needs second-order computations involving the Fisher information. Our rule is first-order, wraps Adam, and takes its scale from the budget. To settle this, we will sweep CPO's $\delta$ across the same orders of magnitude as the learning-rate grid. If CPO's sensitivity to $\delta$ turns out as flat as ours is to learning rate, the contribution narrows to a cheaper first-order version, and we will report it that way.
Why not just clip gradients, or use IPO with Adam? Clipping caps how far a step moves but ignores how close the policy is to the budget. On superlinear objectives, clipped SGD still diverged at large step sizes (Taming SGD). For IPO, LB-SGD's comparison reports that it needs hand-tuned step sizes to avoid violations during training. Both will appear in the learning-rate-sweep plots, which test these claims directly in our setting.
Why not just wrap the optimizer with Mechanic or MoMo? Those tuners only see reward or loss feedback. In safe regions that feedback carries little information about the boundary, and Mechanic is known to degrade when noise masks its feedback signal. The MoMo-only ablation tests this alternative exactly. If it keeps cost as low as our rule does, the cost cap is unnecessary.
Proposed Experiments
The main experiment is a learning-rate-by-method grid. The environments are SafetyPointGoal1-v0, SafetyPointCircle1-v0, and SafetyHopperVelocity-v1 in Safety-Gymnasium, the four Bullet-Safety-Gym tasks, and Humanoid-Circle. The metrics are final return, cumulative training-time cost, the share of iterations over budget, and the Expected Maximum Consecutive Cost steps (EMCC) measure from Revisiting Safe Exploration. EMCC shows whether violations come as long failures or brief blips. The summary statistic is the width, in decades of learning rate, of the band in which a method satisfies both the return and cost criteria above.
The second experiment uses infeasible starts on Circle2D-1 through Circle2D-4 and on SafetyPointGoal1. We measure steps to feasibility and cost accumulated during recovery, against PPO-Lag, CPPO-PID, and CPO. Barrier methods cannot start from infeasible policies, so they are excluded here.
The third experiment is diagnostic. On every step we log the predicted against realized change in cost. This tells us how often the curvature estimate falls below the true local curvature, which is the condition behind our guarantee.
Three outcomes would decide the proposal. It fails if the learning-rate band is no wider than PAPD's. It fails if the $\hat L_t=0$ ablation performs the same, which would mean the BB curvature contributes nothing. It fails if realized cost overshoots predictions by more than the margin $\kappa\hat\sigma_t$ in a large fraction of steps on the locomotion tasks. Each of these is measurable within the planned compute.
References
- Adaptive Primal-Dual Method for Safe Reinforcement Learning
- A Safe Exploration Approach to Constrained Markov Decision Processes
- MoMo: Momentum Models for Adaptive Learning Rates
- SANIA: Polyak-type Optimization Framework Leads to Scale Invariant Stochastic Algorithms
- Barzilai-Borwein Step Size for Stochastic Gradient Descent
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- IPO: Interior-Point Policy Optimization under Constraints
- Constrained Markov Decision Processes via Backward Value Functions
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- First Order Constrained Optimization in Policy Space
- Taming Stochastic Gradient Descent: Almost Sure Convergence and Saddle-Point Avoidance under $(L_{0},L_{1})$-Smoothness
- Mechanic: A Learning Rate Tuner
- Revisiting Safe Exploration in Safe Reinforcement learning