PID Control for Stable Constrained Reinforcement Learning

This direction is merged into another direction: safe exploration, PID-stabilized Lagrangian optimization. Its last version stays readable and citable here.

Version 1 of 115 papers21 conceptsfrozen 2026-10-04

In constrained reinforcement learning (RL), an agent must maximize a reward while ensuring that a cost function stays below a specific threshold. The standard mathematical tool for this is the Lagrangian relaxation, which converts the constrained problem into an unconstrained one by introducing a penalty term, $\lambda$.

However, training these agents is notoriously unstable. Agents often oscillate between being "too brave" (violating constraints) and "too cowardly" (ignoring rewards). Recent research has identified that this instability stems from how we update the penalty term $\lambda$, and that the solution lies in treating $\lambda$ not just as a mathematical variable, but as a controlled feedback loop.

The Problem: The Lagrangian See-Saw

The classic approach to constrained RL, as used in algorithms like Constrained Policy Optimization, updates the Lagrange multiplier $\lambda$ using gradient ascent. If the agent violates a constraint, $\lambda$ increases to penalize that behavior; if the agent is safe, $\lambda$ decreases.

Mathematically, the update rule looks like this: $$\lambda_{t+1} = \max(0, \lambda_t + \eta (J_C - d))$$ where $J_C$ is the current cost and $d$ is the safety limit.

As noted by Adam Stooke, Joshua Achiam, and Pieter Abbeel in Responsive Safety in Reinforcement Learning by PID Lagrangian Methods, this update rule is functionally identical to an Integral (I) controller in classical control theory. Integral controllers are excellent at eliminating steady-state error, but they are famously prone to overshoot and oscillation.

When the agent hits a wall, the "error" (violation) accumulates in $\lambda$. By the time $\lambda$ is large enough to force the agent to stop, it has accumulated so much "momentum" that it keeps rising even after the agent is safe. This causes the agent to become overly conservative, the error then accumulates in the opposite direction, and the cycle repeats.

The Solution: PID Lagrangian Updates

To fix these oscillations, researchers have turned to the Proportional-Integral-Derivative (PID) controller. By adding Proportional (P) and Derivative (D) terms to the multiplier update, we can stabilize the learning dynamics.

  1. The Integral Term (I): This is the original Lagrangian update. It ensures that, on average, the constraints are met over the long term.
  2. The Proportional Term (P): This reacts immediately to the current violation. If the agent suddenly breaks a rule, the P-term spikes $\lambda$ instantly, providing a "responsive" safety signal before the integral term has time to wind up. Baiyu Peng and Jianyu Chen demonstrate in Separated Proportional-Integral Lagrangian for Chance Constrained Reinforcement Learning that this combination prevents the periodic "limit cycles" seen in fixed-penalty methods.
  3. The Derivative Term (D): This looks at the trend of the violation. If the agent is rapidly approaching a constraint, the D-term increases the penalty in anticipation. Conversely, if the agent is already moving toward a safe state, the D-term reduces the penalty to prevent overshooting.

Stabilizing the Trade-off

This control-theoretic view changes the role of $\lambda$ from a simple weight to a dynamic regulator. In Conservative Distributional Reinforcement Learning with Safety Constraints, Hengrui Zhang and colleagues show that using a PID controller with weighted averaging damps the instability inherent in the reward-vs-safety trade-off.

This stability is critical for real-world applications. For instance, Jinming Xu and Yuan Lin applied a PID-Lagrangian framework to autonomous lane changing in Safe Hybrid-Action Reinforcement Learning-Based Decision and Control for Discretionary Lane Change, achieving a 0% collision rate. By reducing oscillations in the penalty factor, the agent learns a smoother, more reliable policy that doesn't "panic" when near other vehicles.

Generalizing to Deep Learning

The benefits of PID-style updates extend beyond just RL. In On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization, Motahareh Sohrabi and Jose Gallego-Posada introduce $\nu$PI, which generalizes momentum methods into a PI framework. They show that by monitoring the trend of violations through an exponential moving average, the optimizer can "anticipate" constraint satisfaction and reduce penalties before the constraint is fully met, effectively smoothing the optimization landscape.

This shift is part of a broader argument made by Juan Ramirez and Simon Lacoste-Julien in Position: Adopt Constraints Over Penalties in Deep Learning. They suggest that instead of manually tuning fixed penalty weights—which is difficult and often fails in non-convex problems—practitioners should use constrained optimization with PID-based update rules to let the system automatically and stably find the right balance.

Imagination and Planning

The PID-Lagrangian approach also integrates well with model-based RL. In SafeDreamer: Safe Reinforcement Learning with World Models, Weidong Huang and Yaodong Yang use a PID controller to adjust safety weights while the agent "imagines" future trajectories in a latent world model. This allows the agent to dynamically balance task rewards and safety risks in its "head" before ever taking an action, leading to nearly zero-cost safety performance even in complex, vision-based environments.

By treating the Lagrange multiplier as a feedback control problem, these methods transform constrained RL from a shaky balancing act into a robust, responsive system capable of mastering high-stakes tasks.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pid_control_for_stable_constrained_reinf_6535beac,
  title = {PID Control for Stable Constrained Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pid-control-for-stable-constrained-reinforcement-learning/rdr_6535beac9de8422abf7c982e9cd1a5bb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_pid_control_for_stable_constrained_reinf_6535beac_v1,
  title = {PID Control for Stable Constrained Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pid-control-for-stable-constrained-reinforcement-learning/rdr_6535beac9de8422abf7c982e9cd1a5bb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}