Safe Reinforcement Learning via Constrained Optimization

This direction is merged into another direction: safe exploration. Its last version stays readable and citable here.

Version 1 of 122 papers22 conceptsfrozen 2026-10-04

In traditional reinforcement learning (RL), we teach agents by giving them a single number: a reward. If we want a robot to walk quickly but safely, we might give it a positive reward for speed and a negative penalty for falling. This approach, known as reward shaping, is intuitive but notoriously brittle.

If the penalty for falling is too small, the robot might decide that the occasional crash is a price worth paying for high speed. If the penalty is too large, the robot may become "paralyzed" by fear, refusing to move at all to avoid any risk of falling. Finding the "Goldilocks" value for these penalties is a labor-intensive "black art" that requires constant human intervention.

A growing body of research suggests a more principled alternative: Constrained Reinforcement Learning (CRL). Instead of merging safety and performance into a single scalar reward, CRL treats them as distinct objectives. The goal becomes a constrained optimization problem: maximize the reward subject to the requirement that safety costs stay below a specific budget.

The Failure of Manual Penalty Tuning

The fundamental problem with reward shaping is that the optimal balance between task performance and safety is rarely known in advance. As Santiago Paternain and colleagues point out in Safe Policies for Reinforcement Learning via Primal-Dual Methods, manual tuning is often impractical because the relationship between a penalty weight and the resulting behavior is non-linear and environment-dependent.

This difficulty extends beyond robotics. In the alignment of Large Language Models (LLMs), Juntao Dai and the team behind Safe RLHF: Safe Reinforcement Learning from Human Feedback found that fixed penalties create a "brittle trade-off." A model might become helpful but toxic, or safe but entirely evasive, refusing even harmless requests because the safety penalty is too heavy-handed.

Automating Safety with Primal-Dual Methods

To solve this, researchers use the Lagrangian method. Instead of a human picking a fixed penalty weight, the algorithm treats the penalty as a "Lagrange multiplier" that evolves during training.

If the agent violates safety constraints, the multiplier (the penalty) automatically increases, forcing the agent to prioritize safety. Once the agent is safe, the multiplier may decrease to allow the agent to focus on maximizing rewards. Santiago Paternain proved that this primal-dual approach can achieve near-zero loss in optimality, effectively finding the best possible performance that still respects the safety budget.

This decoupling is powerful. In Evaluation of Constrained Reinforcement Learning Algorithms for Legged Locomotion, Joonho Lee and Marco Hutter demonstrated that algorithms like N-P3O allow legged robots to respect physical limits—like joint torque and velocity—more stably than traditional reward shaping.

Smoothing the Path to Safety

One challenge with constraints is that they are often "hard" (e.g., you either crashed or you didn't), which creates jagged, difficult-to-optimize landscapes for neural networks.

Researchers have introduced several mechanisms to make these constraints easier to navigate:

  1. Log Barrier Functions: Baohe Zhang and Joschka Bödecker in Constrained Reinforcement Learning with Smoothed Log Barrier Function use "smoothed" barriers that provide a gradient for the agent to follow before it actually hits a constraint, enabling safer sim-to-real transfer.
  2. Failure Surrogates: In ConstrainedZero: Chance-Constrained POMDP Planning Using Learned Probabilistic Failure Surrogates and Adaptive Safety Constraints, Robert J. Moss and Mykel J. Kochenderfer use a dedicated neural network head to predict the probability of failure. This allows the agent to plan paths that stay within a specific "safety budget."
  3. The Minmax Penalty: Geraud Nangue Tasse and Benjamin Rosman proposed ROSARL, which calculates a theoretically grounded penalty based on the environment's "controllability," ensuring the agent avoids unsafe states without needing any human-tuned costs.

Real-Time Regulation and Instincts

While optimization helps during training, some methods focus on safety during the act of learning itself. Arthicha Srisuchinnawong and Poramate Manoonpong developed ROGER, a system that dynamically adjusts reward gains based on how close a robot is to a safety boundary. This allowed a 60-kg quadruped to learn to walk in the real world with zero falls during the entire training process, as detailed in Gain Tuning Is Not What You Need.

Similarly, Djordje Grbic and Sebastian Risi proposed Instinct-Regulated Reinforcement Learning (IR2L). This approach uses a pre-trained "instinct" network that acts as a safety guardrail, overriding dangerous exploratory actions before they happen.

Why It Works: The Viability Kernel

At its core, this research direction is about identifying the viability kernel—the set of states from which it is possible to remain safe indefinitely. As Pierre-François Massiani and Sebastian Trimpe prove in Safe Value Functions, there exists a finite penalty magnitude that is "just enough" to keep an agent within this kernel.

By moving away from manual reward shaping and toward constrained optimization, the field is shifting from trial-and-error tuning toward a rigorous engineering discipline where safety is a requirement, not just a suggestion.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_safe_reinforcement_learning_via_constrai_c3cbdcec,
  title = {Safe Reinforcement Learning via Constrained Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/safe-reinforcement-learning-via-constrained-optimization/rdr_c3cbdcecc11f46b788b28b92d0685296},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_safe_reinforcement_learning_via_constrai_c3cbdcec_v1,
  title = {Safe Reinforcement Learning via Constrained Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/safe-reinforcement-learning-via-constrained-optimization/rdr_c3cbdcecc11f46b788b28b92d0685296/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}