In standard Reinforcement Learning (RL), the objective is simple: maximize a cumulative reward signal. However, when deploying agents in the real world—such as autonomous vehicles or industrial robots—maximizing rewards is not enough. We must also satisfy safety constraints, such as avoiding collisions or staying within mechanical limits. This is the domain of Constrained Markov Decision Processes (CMDPs).
The most common way to solve CMDPs is through Lagrangian relaxation, which converts the hard constraint into a penalty term in the reward function. While mathematically elegant, this approach frequently fails in practice. Researchers have identified a recurring problem: optimization instability. Instead of converging to a safe, high-performing policy, agents often oscillate wildly between being "too risky" and "too cautious," a phenomenon that prevents reliable deployment in safety-critical systems.
The Seesaw Effect of Lagrangian Methods
The core of the instability lies in the dual optimization process. In a typical setup, a "primal" agent tries to maximize rewards while a "dual" variable (the Lagrange multiplier) adjusts the penalty for safety violations.
As Haoran Xu and Xianyuan Zhan note in their work on Constraints Penalized Q-Learning (CPQ), these dynamics often become unstable. When the agent violates a constraint, the multiplier increases, making the penalty so severe that the agent abandons the task to stay safe. Once the agent is safe, the multiplier drops, and the agent aggressively pursues rewards again, leading to another violation.
This "seesaw effect" is not just a tuning issue; it is often structural. Jaeyoung Lee and Krzysztof Czarnecki argue in Recursive Constraints to Prevent Instability in Constrained Reinforcement Learning that the constrained Bellman operator may lack a fixed point, leading to "policy chattering" where the agent endlessly cycles between different strategies.
Damping Oscillations with Control Theory
To combat this instability, researchers have begun treating the update of the Lagrange multiplier as a control problem. In Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, Adam Stooke, Joshua Achiam, and Pieter Abbeel propose using PID control to update the penalty weight.
By incorporating derivative and integral terms—rather than just the current violation magnitude—the PID controller can anticipate when the agent is approaching a constraint and dampen the multiplier's response. This prevents the "overshoot" that causes agents to become overly conservative, leading to much smoother learning trajectories and more robust hyperparameters.
Gradient Manipulation and Trust Regions
Another perspective focuses on the conflict between the reward gradient and the safety gradient. When these two vectors point in opposite directions, the optimization step can become erratic.
Bilgehan Sel and colleagues introduced Projection Constraint-Rectified Policy Optimization (PCRPO), which uses gradient manipulation to resolve these conflicts. By implementing a "soft switching" mechanism and a slack-based buffer, the algorithm ensures that updates to the policy do not inadvertently destroy safety progress while chasing rewards.
Similarly, Dohyeong Kim and Songhwai Oh developed TRC (Trust Region Conditional Value at Risk). By constraining the update within a trust region and focusing on "tail-end" risks (CVaR), they provide a differentiable way to ensure the agent stays within a safe manifold throughout the entire learning process, rather than just at the end of training.
Decoupling Safety from Task Performance
Some researchers argue that the instability stems from trying to solve two different problems with one objective. David Henry Mguni and the team behind DESTA frame safe RL as a two-player game of intervention. In this framework, a Safety Agent has the power to override a Task Agent. By decoupling the objectives, the system avoids the mathematical instability of a single coupled objective, allowing the task agent to learn freely while the safety agent maintains a hard boundary.
A similar intuition drives Feasible Policy Iteration (FPI) by Jingliang Duan et al. FPI introduces a "feasibility function" that provides theoretical guarantees for monotonic safety expansion. Instead of oscillating, the agent's safe operating region grows geometrically, ensuring that once a policy is safe, it stays safe.
The Path to Convergence
The shift in this research area is moving away from simple penalty-based methods toward frameworks that provide formal guarantees. Whether through optimistic gradient updates for last-iterate convergence or contraction theory to bake stability directly into neural architectures, the goal is the same: transforming Safe RL from a fragile balancing act into a stable, predictable engineering discipline. By addressing the underlying causes of optimization oscillation, these methods pave the way for RL agents that can be trusted in the real world.