Stabilizing Primal-Dual Dynamics in Constrained Reinforcement Learning

239 concepts168 papers2 proposals

In the effort to deploy reinforcement learning (RL) in safety-critical domains like autonomous driving or robotic surgery, researchers often model the problem as a Constrained Markov Decision Process (CMDP). The goal is to maximize a reward signal while ensuring that a cost signal—representing safety violations—remains below a specific threshold.

The standard mathematical tool for this is the Lagrangian multiplier. By augmenting the reward objective with a weighted penalty for safety violations, the problem is transformed into a min-max game: the "primal" agent tries to maximize rewards, while the "dual" multiplier increases the penalty whenever constraints are breached. However, as Zuxin Liu et al. and Adam Stooke et al. observe, this approach frequently suffers from severe training instability, characterized by wild oscillations and overshooting.

The Seesaw Effect of Competing Gradients

The core of the instability lies in the competing gradients of the two objectives. When an agent discovers a high-reward but unsafe behavior, the safety constraint is violated, causing the Lagrangian multiplier to rise. This increases the penalty, eventually forcing the agent to abandon the behavior. However, as Chen Tessler et al. note, this often leads to a "seesaw" effect: the agent over-corrects for safety at the expense of performance, only to later sacrifice safety again to regain lost rewards.

Tianqi Zheng et al. provide a geometric intuition for this failure. When formulated in occupancy measure space, the Lagrangian objective becomes a bilinear function. Standard gradient descent-ascent algorithms are notoriously poor at solving such structures; instead of converging to the global optimum (the saddle point), the optimization dynamics tend to produce stable orbits around it. This results in "policy oscillation," where the agent never settles on a single, reliable policy, but instead cycles between risk-seeking and risk-averse behaviors.

Why Convergence is Elusive

This instability is not merely a nuisance; it has fundamental theoretical and practical consequences:

  • Last-Iterate Failure: Many algorithms only guarantee that the average of all policies visited during training will satisfy constraints. As Ted Moskovitz et al. argue, this is useless for real-world deployment, where we need the final checkpoint to be safe.
  • Hyperparameter Sensitivity: The inter-dependency between the policy's learning rate and the value of the multipliers makes tuning extremely difficult. Weiqin Chen et al. point out that a learning rate that works when the penalty is small may become unstable as the multiplier grows.
  • Duality Gaps: While Santiago Paternain et al. proved that CMDPs have a zero duality gap in the limit, Shaocong Ma shows that this assumption often fails under model uncertainty, leading to policies that are either sub-optimal or fundamentally unsafe.

Emerging Strategies for Stability

To move beyond these oscillations, researchers have proposed several mechanisms to dampen the primal-dual dynamics:

PID Control for Multipliers Instead of treating the multiplier update as a simple gradient step, Adam Stooke et al. treat it as a control problem. By applying Proportional-Integral-Derivative (PID) control to the multiplier, they can reduce overshoot and make the safety mechanism more responsive to sudden spikes in cost, leading to smoother convergence.

Optimistic Updates To achieve "last-iterate convergence," Ted Moskovitz et al. and Dongsheng Ding et al. utilize optimistic gradient updates. By incorporating information from previous iterations, these methods effectively "look ahead" to counteract the rotational tendencies of the optimization flow, pushing the agent toward the saddle point rather than around it.

Decoupling and Probabilistic Inference Some researchers argue the conflict is best solved by separating the objectives entirely. Zuxin Liu et al. propose Constrained Variational Policy Optimization (CVPO), which reframes the problem as probabilistic inference. By using an Expectation-Maximization framework, they decouple reward maximization from the policy update, preventing the competing gradients from interfering with each other directly.

Interior-Point Methods Borrowing from classical optimization, Nico Scherf introduces the "central path" to RL. By using log-barrier functions instead of hard Lagrangian penalties, the optimization process is guided through the interior of the feasible region, avoiding the erratic jumps that occur when an agent hits a safety boundary.

By addressing the underlying dynamics of the primal-dual interaction, these methods aim to transform Constrained RL from a volatile balancing act into a stable, reliable framework for safety-critical AI.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 181 papers
  2. Version 1 · 2026-10-04 · 278 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_lagrangian_optimization_instability_20faa724,
  title = {Lagrangian optimization instability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lagrangian-optimization-instability/rdr_20faa724138446818c78e34f2cce0fb8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_lagrangian_optimization_instability_20faa724_v2,
  title = {Lagrangian optimization instability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lagrangian-optimization-instability/rdr_20faa724138446818c78e34f2cce0fb8/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake