Abstract
Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent's policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research.
Introduction
Reinforcement Learning (RL) has demonstrated remarkable success in complex tasks, but its application in real-world scenarios is often hindered by safety concerns. Traditional RL focuses on maximizing cumulative rewards, which can lead to "reckless" behavior—such as a drone crashing or an LLM generating harmful content—if safety is not explicitly modeled. Safe RL addresses this by incorporating constraints into the optimization process. However, the field has historically suffered from a lack of standardization, with various mathematical definitions of safety competing for dominance. This paper provides a systematic taxonomy and theoretical unification of these diverse constraint formulations, creating a "map" for a previously fragmented landscape.
A Taxonomy of Safety Constraints
The authors identify seven distinct safety constraint formulations, organized along two primary axes:
- Expectation-based ($\mathbb{E}_\pi$) vs. Probability-based ($\mathbb{P}_\pi$): Expectation-based constraints focus on average safety over many trials, while probability-based constraints aim for "almost sure" safety, demanding that the agent avoids unsafe states with high certainty.
- Cumulative vs. Instantaneous Costs: Cumulative constraints consider the long-term sum of costs over an episode, whereas instantaneous constraints enforce safety at every individual time step.
By categorizing safety into these quadrants, the survey helps engineers identify which mathematical language best fits their specific real-world problem.
Theoretical Unification: The "Master" Formulations
A key contribution of this work is the introduction of Identical or More General Safe RL (IoMG-SafeRL). This theoretical framework identifies "master" problems that are flexible enough to represent or approximate other formulations. Through three primary theorems, the authors establish a hierarchy:
- Problem 4 (Expected Instantaneous): Proved to be a master formulation for standard Constrained MDPs (Problem 1) and state constraints (Problem 2). It also acts as a "safe" approximation for complex Joint Chance Constraints common in control theory.
- Problem 7 (Almost Surely Instantaneous): Established as the master formulation for all "almost surely" safety constraints (Problems 5 and 6).
By developing algorithms that can solve these general formulations, researchers can theoretically address a wide array of specific safety requirements across the literature.
The Safety-Performance Trade-off
The survey highlights a fundamental tension: higher safety guarantees often come at the cost of performance. Probability-based models ($\mathbb{P}_\pi$) provide the most rigorous safety guarantees but significantly handicap the agent's ability to collect rewards compared to expectation-based models. Furthermore, the paper demonstrates that while some hard-to-solve problems are computationally daunting, they can often be "conservatively approximated" by simpler additive cost models, making them more tractable for practical deployment.
Algorithmic Mapping and Resources
Rather than proposing a single new algorithm, the authors map dozens of existing methods—such as CPO, RCPO, and PCPO—to the specific formulations they were designed to solve. They argue that the most common mistake in Safe RL is picking an algorithm before fully defining the safety requirement. To assist practitioners, the paper points to several high-quality Open Source Software (OSS) libraries and benchmarks, including:
- OmniSafe / SafePO: Modern, actively maintained libraries for safe RL.
- Safety Starter Agents (SSA): A foundational repository from OpenAI.
- FSRL / OSRL: Specialized libraries for federated and offline safe RL.
Conclusion and Future Directions
This survey bridges the gap between theoretical safety definitions and practical algorithm selection. While it clarifies how to define safety numerically, it acknowledges that satisfying these constraints during the training phase (online) remains a massive challenge. Looking forward, the authors identify Safe RLHF (Reinforcement Learning from Human Feedback) as a critical frontier. In this context, safety is not just a numerical threshold but a complex human preference that must be balanced against model helpfulness, particularly in the development of Large Language Models.
Summary updated