In reinforcement learning (RL), we typically task an agent with maximizing a scalar reward signal. However, real-world deployment often requires satisfying strict safety or resource boundaries. We model these scenarios as Constrained Markov Decision Processes (CMDPs). For instance, an autonomous vehicle must maximize its speed and efficiency while keeping its collision probability below a strict threshold.
The standard mathematical tool for solving CMDPs is the Lagrangian dual formulation. We convert the constrained problem into an unconstrained one by introducing a penalty term governed by a Lagrange multiplier, $\lambda$:
$$\max_{\pi} \min_{\lambda} \mathcal{L}(\pi, \lambda) = R(\pi) - \lambda (C(\pi) - V_0)$$
where $R(\pi)$ is the expected reward, $C(\pi)$ is the constraint cost, and $V_0$ is the allowed safety budget. To find the saddle point of this objective, algorithms traditionally employ primal-dual gradient descent: the policy $\pi$ is updated to maximize $\mathcal{L}$, while $\lambda$ is updated via gradient ascent to penalize constraint violations.
The Problem: The Lagrangian Seesaw
While elegant in theory, standard primal-dual optimization notoriously destabilizes reinforcement learning. When trained with deep neural networks, the agent's policy and the Lagrange multiplier frequently enter a destructive cycle of wild oscillations.
When the agent violates a safety constraint ($C(\pi) > V_0$), $\lambda$ begins to grow. However, because neural network policy updates are stochastic and incremental, it takes many iterations for the policy to respond to this rising penalty. By the time the policy finally adapts, $\lambda$ has grown excessively large. The agent, terrified of the massive penalty, stops seeking rewards entirely and adopts an overly conservative behavior.
Once the agent is safe ($C(\pi) < V_0$), $\lambda$ begins to decrease. Again, due to the delayed response of the policy, $\lambda$ drops too low before the agent notices. The agent forgets its safety constraints, aggressively pursues rewards, and begins crashing again. This "seesaw" effect severely degrades training efficiency and leads to frequent, dangerous constraint violations during the learning process.
The Insight: Dual Updates as an Integral Controller
To understand why this oscillation happens—and how to fix it—we can look at the standard update rule for the Lagrange multiplier through the lens of control theory. The classic gradient ascent update for $\lambda$ at step $t$ is:
$$\lambda_{t+1} = \max(0, \lambda_t + \alpha (C(\pi_t) - V_0))$$
If we define the constraint violation as an error signal, $e_t = C(\pi_t) - V_0$, and unroll this recursive formula over time, we reveal a telling mathematical identity:
$$\lambda_{t+1} = \alpha \sum_{i=0}^{t} e_i$$
As recognized by Adam Stooke et al. in their foundational paper Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, standard dual gradient descent is exactly equivalent to a pure Integral (I) controller.
In classical control theory, it is well known that pure integral control applied to dynamical systems with delays or high-order lag inherently causes overshoot and instability. The "integral wind-up" accumulates past errors, forcing the controller to over-correct long after the error has begun to decrease.
The Solution: Designing a PID Lagrangian Multiplier
Once the connection between dual optimization and control theory is established, the solution becomes clear: we can stabilize the optimization loop by upgrading the pure integral update to a full Proportional-Integral-Derivative (PID) controller.
By treating the adjustment of $\lambda$ as a control loop where the target is to keep the error $e_t = 0$, we can introduce Proportional (P) and Derivative (D) terms to counteract the lag of the Integral (I) term:
$$\lambda_t = K_p e_t + K_i \sum_{i=0}^t e_i + K_d (e_t - e_{t-1})$$
Each component plays a distinct role in stabilizing the agent's learning trajectory:
- The Proportional Term ($K_p$) provides an immediate response to the current magnitude of the violation. If the agent suddenly enters a highly dangerous state, the P-term immediately spikes the penalty, forcing the policy to react without waiting for the integral term to slowly accumulate. As shown by Baiyu Peng et al. in their Separated Proportional-Integral Lagrangian (SPIL) framework, balancing immediate violations with accumulated error prevents the periodic oscillations typical of fixed-penalty or pure-integral methods.
- The Integral Term ($K_i$) retains the long-term memory of the system, ensuring that steady-state constraint violations are eventually driven down to zero.
- The Derivative Term ($K_d$) acts as a predictive dampener by measuring the rate of change of the violation. If the agent is rapidly heading toward danger, the D-term increases the penalty preemptively. Conversely, if the agent is actively correcting its behavior and moving back toward the safe region, the D-term senses this positive trend and eases off the brakes, preventing the massive overshoot that causes the policy to freeze up.
This control-theoretic framework dramatically smooths out the penalty coefficient, preventing the agent from alternating between extreme caution and reckless risk-taking.
Alternative Perspectives: Damping and Augmented Lagrangians
The intuition behind PID-stabilized optimization is mirrored in several parallel lines of research that approach the problem from optimization theory rather than control loops.
Augmented Lagrangians and Quadratic Penalties
Instead of modifying the update rule directly, one can modify the objective function itself. Adrian Müller et al. in Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes and Juntao Dai et al. in Augmented Proximal Policy Optimization for Safe Reinforcement Learning utilize an Augmented Lagrangian approach. They inject a quadratic penalty term directly into the objective:
$$\mathcal{L}_{\text{aug}}(\pi, \lambda) = R(\pi) - \lambda (C(\pi) - V_0) - \frac{\rho}{2} \|C(\pi) - V_0\|^2$$
This quadratic deviation term acts as an instantaneous stabilizer. It penalizes large deviations from the feasible boundary, providing continuous feedback that mathematically mimics the stabilizing effects of a proportional control term.
Explicit Multiplier Damping
Another direct approach is to introduce an explicit damping factor into the dual update rule. Longfei Yue et al. and Dongsheng Ding et al. demonstrate that integrating a damping term or regularizing the policy gradient trajectory prevents the safety coefficient from oscillating wildly. This ensures smooth, last-iterate convergence to safe and optimal control strategies rather than relying on the average of a highly unstable trajectory.
Real-World Impact and Applications
Stabilizing the Lagrangian optimization loop has unlocked the practical deployment of constrained RL across several high-stakes domains where training-time failures are unacceptable.
Autonomous Driving and Robotics
In autonomous driving, standard RL algorithms cannot be trained on real roads because their oscillatory exploration phase causes frequent collisions. By utilizing a PID-controlled Lagrangian optimization loop, Zhenghao Peng et al. developed an expert-guided policy optimization (EGPO) mechanism that achieves near-zero training-time collisions, allowing safe and steady policy convergence.
Eliminating "Action Shaking"
In continuous control tasks, unstable constraint enforcement often manifests as "action shaking"—jittery, high-frequency control signals that would destroy physical hardware. Guangyuan Zou et al. integrated a PID controller to dynamically tune multiple Lagrange multipliers simultaneously. This framework treats action smoothness as a strict constraint, eliminating jitter without sacrificing the agent's ability to maximize rewards.
Inverse Constraint Learning
When safety boundaries are not known a priori, they must be learned from expert demonstrations. Konwoo Kim et al. integrated PID-based Lagrangian methods into Multi-Task Inverse Constraint Learning (MT-ICL). Stabilizing the dual variables proved essential here: without it, the algorithm could not distinguish between an agent trying to avoid a true safety hazard and an agent caught in a temporary, optimization-induced panic phase.
Broader Optimization Contexts
The necessity of stabilizing the Lagrangian objective extends beyond standard policy optimization. For instance, Weiqin Chen et al. combined PID updates with adaptive learning rates to enhance hyperparameter robustness. In off-policy evaluation (OPE), Mengjiao Yang 0001 et al. showed that regularizing the Lagrangian with redundant constraints improves convergence when using neural function approximators. Even in quantum control architectures, Jan Ole Ernst et al. leveraged physics-constrained optimization to prevent chaotic control signals from creating computational bottlenecks in parallelized simulation loops.
Recognizing the widespread utility of these control-theoretic stabilizers, modern benchmarking suites like Safety Gymnasium, developed by Jiaming Ji et al., now include PID Lagrangian methods as standard, robust baselines for evaluating any new safe reinforcement learning algorithm.
Key Foundational Work
Safe Driving via Expert Guided Policy Optimization (2021) — contributed 2 concept elements to this direction.
Responsive Safety in Reinforcement Learning by PID Lagrangian Methods (2020) — contributed 2 concept elements to this direction.
Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning (2021) — contributed 1 concept elements to this direction.