Abstract
Safety is critical when applying reinforcement learning (RL) to real-world problems. As a result, safe RL has emerged as a fundamental and powerful paradigm for optimizing an agent’s policy while incorporating notions of safety. A prevalent safe RL approach is based on a constrained criterion, which seeks to maximize the expected cumulative reward subject to specific safety constraints. Despite recent effort to enhance safety in RL, a systematic understanding of the field remains difficult. This challenge stems from the diversity of constraint representations and little exploration of their interrelations. To bridge this knowledge gap, we present a comprehensive review of representative constraint formulations, along with a curated selection of algorithms designed specifically for each formulation. In addition, we elucidate the theoretical underpinnings that reveal the mathematical mutual relations among common problem formulations. We conclude with a discussion of the current state and future directions of safe reinforcement learning research.
Introduction
Reinforcement Learning (RL) has achieved significant success in simulated environments, yet its deployment in high-stakes real-world applications—such as autonomous driving, surgical robotics, and large language model fine-tuning—requires rigorous safety guarantees. Safe RL has emerged to address this need by optimizing policies under specific safety constraints. However, the field is currently fragmented by a diverse array of mathematical formulations, ranging from state-based avoidance to cumulative cost minimization. This lack of a unified framework makes it difficult for practitioners to select appropriate algorithms or understand how different safety definitions relate to one another. This work provides a systematic taxonomy and theoretical mapping of these formulations within the framework of Constrained Markov Decision Processes (CMDPs).
Theoretical Framework and Interrelations
The core contribution of this survey is the categorization of seven representative constraint formulations and the derivation of their mathematical interrelations. These relations are defined through three primary notions:
- Transformability: The ability to mathematically convert one problem type into another.
- Generalizability: Identifying if one formulation acts as a superset of another.
- Conservative Approximation: Determining if satisfying a stricter constraint (Problem B) guarantees the satisfaction of a more lenient one (Problem A).
Expectation-based vs. Almost-surely Constraints
Constraints are broadly divided into those based on Expectation ($\mathbb{E}_\pi$), which focus on average safety performance, and those based on Almost-surely ($\mathbb{P}_\pi$) criteria, which target "hard" safety guarantees where failures must be avoided with high probability.
As illustrated in the relational map above, Problem 4 (Expected Instantaneous Constraints) emerges as a "master" formulation. Through the use of a "remaining safety budget" (Lemma 1), cumulative constraints (Problem 1) can be transformed into instantaneous ones. Furthermore, the analysis reveals that "Joint Chance Constraints" (Problem 3), while often computationally intractable, can be safely approximated by simpler additive cost constraints.
The Algorithmic Landscape
Beyond theoretical mapping, this survey categorizes existing Safe RL algorithms based on the specific problem formulations they address. This includes foundational methods like Constrained Policy Optimization (CPO) and more recent developments like Simmer RL.
The table above serves as a comprehensive reference for the current state of the art, detailing whether algorithms provide optimality or safety guarantees during training versus only after convergence. It also highlights the availability of Open Source Software (OSS) libraries, such as OmniSafe, SafePO, and OSRL, which facilitate the implementation of these diverse formulations.
Key Insights and Future Directions
A significant finding of this work is the identification of Identical or More General Safe RL (IoMG-SafeRL) problems, specifically Problem 4 and Problem 7 (Almost Surely Instantaneous Constraints). These formulations are theoretically powerful enough to represent or approximate most other safety criteria, suggesting that future research should prioritize developing robust solvers for these "master" problems.
Despite progress, several challenges remain. Stricter safety formulations often impose a "safety tax," leading to lower cumulative rewards. Future work should explore the use of Large Language Models (LLMs) to bridge the gap between human safety preferences and these rigorous mathematical formulations, as well as expanding the scope to include formal logic-based constraints.
Summary updated