A safe reinforcement learning agent is usually asked to keep several costs under budget at once: hits on hazards, collisions with objects, excess speed. The standard tools handle each constraint with a separate learned multiplier, a "price" on violation that rises when the agent overspends and falls when it underspends. These prices oscillate. The agent swings between reckless violation and over-cautious freezing, and with several constraints the swings interact. A second, less discussed problem is that the agent only sees noisy estimates of its costs. Near the budget, the estimated sign of "am I violating?" flips from batch to batch, so any rule that switches on violation chatters exactly where the optimal policy has to live. Fan et al. prove that this region, the constraint boundary, is where the optimal policy lies whenever the unconstrained optimum is unsafe (Boundary-Seeking Policy Gradient for Safe Reinforcement Learning).
We propose to replace the multipliers with a single non-smooth exact penalty, the largest constraint excess clipped at zero, and to smooth only that penalty with its Moreau envelope. The Moreau envelope is the best value reachable by a nearby point when moving away costs a quadratic price; it turns a kinked function into one with a gradient that changes at a controlled rate. The smoothing width is tied to the noise in the cost estimates and shrinks over training. The resulting policy update needs no dual variables and no second-order matrices, and it gives each constraint a weight computed in closed form from the current excesses. Two practical claims follow, and both can be tested. The update should remove chattering at constraint intersections and at a noisy boundary. And the envelope's gradient norm should give a usable certificate of near-stationarity for a constrained problem that classical gradients cannot measure.
Core Claim
Let $J_R(\theta)$ be the expected return of a policy with parameters $\theta$, and let $z(\theta) \in \mathbb{R}^K$ be the vector of estimated cost excesses, with entries $z_k = \hat J_{C_k}(\theta) - \epsilon_k + \kappa\mu$. Here $\epsilon_k$ is the budget for constraint $k$, $\mu > 0$ is the smoothing width, and $\kappa \ge 0$ sets an inner safety margin. With $g(z) = \max(0, z_1, \dots, z_K)$, we minimize
$$F_\mu(\theta) = -J_R(\theta) + \rho\, M_\mu g\big(z(\theta)\big), \qquad \nabla F_\mu(\theta) = -\nabla J_R(\theta) + \rho \sum_{k=1}^K w_k \nabla J_{C_k}(\theta),$$
where $\rho$ is a fixed penalty scale and $w = (z - \mathrm{prox}_{\mu g}(z))/\mu$.
Because $g$ is convex and piecewise linear, its proximal map can be computed exactly by a sort-and-threshold step. The weights $w$ are nonnegative, sum to at most one, and lie in the subdifferential of $g$ at the proximal point. This is the general identity reviewed in Anderson Acceleration of Proximal Gradient Methods: an envelope-gradient norm of $\eta$ certifies a point within $\mu\eta$ that has a subgradient of norm at most $\eta$.
The objective is a smooth map composed with a convex non-smooth outer function. That is the composite class for which Böhm and Wright prove that decreasing the smoothing width in epochs reaches $\epsilon$-stationarity in $O(\epsilon^{-3})$ iterations with exact gradients (Variable Smoothing for Weakly Convex Composite Functions). We reuse their method and add what RL requires. The excess vector $z$ is a noisy Monte Carlo estimate. Since $\nabla M_\mu g$ is $1/\mu$-Lipschitz, a cost-estimate error of size $\sigma$ perturbs the weights by at most $\sigma/\mu$. This motivates a schedule in which $\mu_k$ shrinks no faster than a constant multiple of the running standard error of the cost estimates.
Honesty about the single-constraint case shapes the whole proposal. With $K = 1$, the weight reduces to $\mathrm{clip}(z/\mu, 0, 1)$, a saturating multiplier proportional to the current violation. That is close to proportional control of a Lagrange multiplier. The proposal's content therefore rests on three things. The first is the multi-constraint case, where the proximal map spreads weight among near-active constraints in a way no per-constraint controller does. The second is the noise-coupled schedule. The third is the stationarity certificate. If multi-constraint experiments show no gain over per-constraint PID multipliers, the idea is not worth pursuing.
Relation to Prior Work
Moreau envelopes have entered RL before, but not for constraints. MEMRL smooths task returns to obtain first-order meta-policy updates and handles no costs (On First-Order Meta-Reinforcement Learning with Moreau Envelopes). The LQR variant of that work is likewise unconstrained (A Moreau Envelope Approach for LQR Meta-Policy Estimation).
On the constrained-optimization side, two papers smooth the Lagrangian itself. iMELa does so under error-bound conditions, but it assumes convex constraints, a polytope domain and deterministic gradients, and it names the stochastic case as open (Inexact Moreau Envelope Lagrangian Method for Non-Convex Constrained Optimization under Local Error Bound Conditions on Constraint Functions). A related method is restricted to linear constraints (Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints).
Our design deliberately avoids the hardest part of those methods, an inexact inner proximal solve over non-convex cost functions of a neural policy. We smooth only the outer max-hinge, whose proximal map is exact and cheap. The non-convexity stays in the smooth inner map, where the composite analysis of Böhm and Wright allows it. The stochastic analysis of weakly convex objectives by Alacaoglu et al. (Convergence of adaptive algorithms for constrained weakly convex optimization) and by Zhu et al. (Stochastic Gradient Methods under Heavy-Tailed Noises in Weakly Convex Optimization) supplies tools for the noisy-gradient part. Neither addresses noise inside the argument of the non-smooth function, which is the RL-specific difficulty here.
On the safe RL side, three papers motivate the proposal by showing the gap. BSPG motivates the focus on the boundary and explicitly leaves multiple intersecting constraints as future work. The documented oscillation of Lagrangian methods motivates the dual-free design (Constrained Policy Optimization, IPO: Interior-Point Policy Optimization under Constraints). The penalty-tuning trap reported in Safety Gymnasium also motivates it: too small a penalty buys violations, too large a penalty freezes the agent.
Other work constrains the design. IPO's log barrier is undefined once a noisy step crosses the boundary, so any smooth surrogate must stay well-defined in the infeasible region; the envelope does. FDPI shows that safer policies starve the learner of violation data (Breaking Safety Paradox with Feasible Dual Policy Iteration). We do not claim to solve that problem. The margin $\kappa\mu$ makes the policy approach the boundary from inside rather than learn it by crossing, and we will measure whether this degrades late-training cost estimates. SPG ties safety to the smoothness of the policy class rather than of the cost penalty (Smoothing policies and safe policy gradients). Its curvature bounds could later bound our step sizes, but they play no role in the method itself.
Research Program
We will build the method on the PPO-style actor-critic in SafePO, the library bundled with Safety-Gymnasium. Each constraint gets its own cost critic and its own advantage estimate. Per-constraint policy gradients are combined with the proximal weights $w$ before the clipped surrogate step.
The schedule lowers $\mu$ in epochs, following Böhm and Wright, with a floor of $c\,\hat\sigma$, where $\hat\sigma$ is the batch standard error of the excess estimates. The penalty scale $\rho$ is raised only when violations persist after the smoothing floor is reached. Exact-penalty theory needs $\rho$ to exceed the optimal multipliers; we report how often that adaptation fires and do not hide it.
The analysis has two parts. First, we derive a bias bound for the stochastic envelope gradient in terms of $\sigma/\mu$ and the cost-gradient magnitude. From it we obtain an iteration-complexity statement that reduces to the $O(\epsilon^{-3})$ rate when $\sigma = 0$. Second, we test empirically whether the envelope-gradient norm, logged every iteration, predicts when constraint residuals and returns stop changing. That would make it a usable stopping and model-selection signal.
Baselines
The naive reference points are unconstrained PPO and PPO with a fixed reward penalty summed over cost channels. For the penalty, we tune the coefficient on a grid the same size as the proposal's tuning grid. Unconstrained PPO bounds achievable return. The fixed penalty is the simplest dual-free method, so it tests whether smoothing and scheduling add anything over a constant price.
The strongest existing methods are PPO-Lagrangian and PID-Lagrangian from SafePO, each with one multiplier per constraint. They are the standard first-order answer to multiple constraints, and PID damping is designed precisely against the oscillation we target. The file reports that PPO-Lag achieves a 98% reduction in safety violations relative to PPO at a 45% drop in total reward on Safety-Gymnasium velocity tasks (Safety Gymnasium). Beating it means matching or lowering final per-constraint cost while recovering a substantial part of that lost reward, and cutting cumulative training-time violation.
FOCOPS is the strongest first-order trust-region method in the file. It reports a Humanoid-Circle reward of 1106.1 at cost within the limit of 50, against 671.0 for CPO and 697.5 for TRPO-Lagrangian (First Order Constrained Optimization in Policy Space). On that single-constraint task, the bar is to match 1106.1 within seed variance at the same cost limit. We do not expect to exceed it, because the single-constraint case is nearly a proportional controller. CPO is included as the second-order reference.
BSPG is the closest boundary-tracking method; no scalar result was found on Lacuna for it. We will extend it to multiple constraints in the most natural way, projecting the reward gradient onto the tangent space of all currently active constraints, and run it on the same tasks.
All methods share environments, network sizes, sample budgets, and an equal number of hyperparameter configurations chosen on held-out seeds. Results are reported over ten seeds.
Four ablations isolate the key ingredient. The first removes smoothing ($\mu \to 0$, the raw subgradient of the max-hinge). The second fixes $\mu$ with no schedule. The third replaces the Moreau envelope with a log-sum-exp smoothing of the max at matched gradient-Lipschitz constant. The fourth is a squared-excess penalty, equivalent to an augmented Lagrangian without its multiplier. Each is compared against the full method.
Why not just…?
Why not just use PID-Lagrangian multipliers per constraint? This is the real threat. For a single constraint our weight is nearly a saturating proportional controller, so we expect parity there. The case for the proposal is that independent per-constraint controllers do not coordinate where two constraints are simultaneously near-active. The proximal map of the max couples them, splitting a bounded total weight among them. The multi-constraint experiments below settle this directly. If PID-Lagrangian matches our cumulative violation and residual oscillation at intersections, we drop the claim.
Why not just smooth the max with log-sum-exp? It is also smooth and cheap. However, it assigns positive weight to every constraint, including slack ones. It also penalizes the policy even when every constraint is comfortably satisfied, which biases the policy inward and away from the boundary BSPG shows is optimal. The Moreau envelope is exactly zero below the margin and assigns weight only to near-active constraints. The matched-smoothness ablation tests whether this difference changes final return at equal cost.
Why not just use a quadratic penalty or augmented Lagrangian? A squared penalty is not exact. It satisfies constraints only as its coefficient grows, recreating the tuning trap described in Safety Gymnasium. An augmented Lagrangian reintroduces a dual state that can oscillate. The squared-penalty ablation, and PPO-Lag as the multiplier-based reference, quantify both effects.
Why not just use CPO's trust region, which already guarantees per-step safety? With several linearized constraints, CPO's subproblem becomes infeasible more often and falls back to recovery steps. On a single constraint it already trails FOCOPS (671.0 against 1106.1 on Humanoid-Circle). We include it, but we do not expect it to be the binding competitor.
Proposed Experiments
The main testbed is Safety-Gymnasium. We will use Goal2 and Button2 navigation tasks with the Point and Car bodies, with the hazard and vase (or gremlin) costs split into separate constraint channels with separate budgets. We will also use the velocity-constrained locomotion tasks with a second, contact-based cost channel added. Single-constraint checks are SafetyPointGoal1, the task BSPG uses, and Humanoid-Circle, against the FOCOPS figure.
Metrics are final return, final cost per channel, and cumulative training-time violation, measured as summed positive excess over all training episodes. We also measure oscillation by the number of sign changes in each residual and by residual variance over the last half of training, and we report the fraction of iterations in which two or more constraints are within one standard error of their budgets.
Three checks are decisive. First, on the multi-constraint tasks, the full method must reduce cumulative violation and intersection-region oscillation relative to PID-Lagrangian at equal or better return. Without that result, the proposal reduces to a reparameterized proportional controller. Second, the unsmoothed and fixed-$\mu$ ablations must show more chattering than the scheduled version, with the gap growing as the batch size shrinks and cost noise rises. This would confirm that coupling $\mu$ to estimator noise is the active ingredient. Third, the logged envelope-gradient norm should fall below a threshold no later than the point at which returns and residuals plateau, judged by rank correlation across seeds. We will also check whether the inward margin lowers late-training critic accuracy on violation states, the failure FDPI documents. If it does, the margin is costing what it was meant to save, and we will report that trade-off rather than tune it away.
References
- Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
- Anderson Acceleration of Proximal Gradient Methods
- Variable Smoothing for Weakly Convex Composite Functions
- On First-Order Meta-Reinforcement Learning with Moreau Envelopes
- A Moreau Envelope Approach for LQR Meta-Policy Estimation
- Inexact Moreau Envelope Lagrangian Method for Non-Convex Constrained Optimization under Local Error Bound Conditions on Constraint Functions
- Moreau Envelope Augmented Lagrangian Method for Nonconvex Optimization with Linear Constraints
- Convergence of adaptive algorithms for constrained weakly convex optimization
- Stochastic Gradient Methods under Heavy-Tailed Noises in Weakly Convex Optimization
- Constrained Policy Optimization
- IPO: Interior-Point Policy Optimization under Constraints
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- Smoothing policies and safe policy gradients
- First Order Constrained Optimization in Policy Space