State-wise Safe Reinforcement Learning: A Survey

Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, Changliu Liu

Abstract

Reinforcement Learning (RL) has demonstrated remarkable potential in simulated environments, yet its inherent "trial-and-error" nature poses significant risks in real-world applications. Traditional Safe RL frameworks, such as Constrained Markov Decision Processes (CMDPs), typically focus on cumulative costs—ensuring safety "on average." However, for high-stakes tasks like autonomous driving or robotic surgery, a single violation can be catastrophic. This survey provides a comprehensive overview of State-wise Safe Reinforcement Learning, a subfield dedicated to satisfying "hard" constraints at every individual time step. We categorize existing literature based on safety guarantees, environment assumptions, and enforcement mechanisms, offering a strategic roadmap for deploying RL in safety-critical systems.

The Shift to State-wise Constraints

While standard RL maximizes cumulative rewards, Safe RL introduces constraints. Most existing work utilizes the CMDP framework, where the objective is to keep the expected sum of costs below a threshold. In contrast, State-wise Constrained MDPs (SCMDPs) demand that constraints are satisfied instantaneously and deterministically: $c(s_t, a_t) \leq 0$ for all $t$.

The primary challenge in SCMDPs is balancing strict constraint satisfaction with the need for effective exploration. If an agent is too conservative, it may never discover the optimal policy; if it is too aggressive, it risks irreversible damage.

Methodological Philosophies

The survey classifies state-wise safe RL into two primary philosophies based on when safety is guaranteed:

1. Asymptotic Safety (After Convergence)

These methods prioritize exploration during the training phase, allowing for occasional "unsafe" behaviors to better map the state space. Safety is enforced gradually using soft penalties in the reward function or "safe critics"—value functions that estimate the probability of future constraint violations. The goal is for the policy to eventually converge to a safe region.

2. In-training Safety (Guaranteed Throughout)

Stricter approaches ensure the agent never violates a constraint, even during the earliest stages of learning. This is typically achieved through:

  • Hierarchical Structures: A safety layer or monitor intercepts the RL actor's proposed actions and corrects them if they lead to a violation.
  • Projection and Shielding: Mathematical tools like Control Barrier Functions (CBFs) or Lyapunov functions map unsafe actions back into a predefined "safe set."
  • Safe Exploration: The agent starts with a known safe policy and cautiously expands its searchable area only when safety can be theoretically guaranteed in the new region.
Visualizing safety philosophies: (a) asymptotic safety, (b) in-training safety via projection, and (c) safe exploration through set expansion.

A Taxonomy of Safe RL

To navigate the diverse landscape of algorithms, the authors propose a taxonomy based on five critical dimensions: theoretical guarantees, environment assumptions (e.g., white-box vs. black-box dynamics), safety evaluation methods, and enforcement strategies.

Taxonomy of state-wise safe RL methods categorized by safety guarantees, assumptions, and enforcement mechanisms.

As shown in the taxonomy, "Asymptotic" methods often lack formal theoretical guarantees and rely on learned cost functions. In contrast, "In-training" methods frequently require "white-box" knowledge—such as exact physics equations—to provide rigorous proofs of safety.

Key Insights and Trade-offs

  • The Safety-Performance Trade-off: There is an inherent tension between exploration and safety. Conservative "in-training" methods may settle for sub-optimal rewards because they avoid potentially high-reward but risky areas.
  • Knowledge vs. Scalability: Formal safety guarantees (like those from Reachability analysis) often struggle to scale to high-dimensional systems, such as robots with many degrees of freedom.
  • Reward Distortion: Safety filters can "distort" the reward signal. If an RL agent's action is constantly overridden by a safety layer, the agent may struggle to understand the true consequences of its decisions, hindering the learning process.

Future Directions

The survey identifies several hurdles for the next generation of Safe RL:

  1. Reducing Reliance on Perfect Models: Most guaranteed-safe methods require highly accurate world models, which are rarely available in complex real-world scenarios.
  2. End-to-End Safety: Developing methods that provide asymptotic safety guarantees without requiring separate, hand-coded safety filters remains a major goal.
  3. Handling Model Mismatch: Future research must focus on the continual correction of dynamics models to ensure safety even when the agent's internal model of the world is slightly inaccurate.

By bridging the gap between formal control theory and modern deep RL, state-wise safe RL paves the way for autonomous systems that are both intelligent and reliably safe.

Summary updated

Abstract

Despite the tremendous success of Reinforcement Learning (RL) algorithms in simulation environments, applying RL to real-world applications still faces many challenges. A major concern is safety, in another word, constraint satisfaction. State-wise constraints are one of the most common constraints in real-world applications and one of the most challenging constraints in Safe RL. Enforcing state-wise constraints is necessary and essential to many challenging tasks such as autonomous driving, robot manipulation. This paper provides a comprehensive review of existing approaches that address state-wise constraints in RL. Under the framework of State-wise Constrained Markov Decision Process (SCMDP), we will discuss the connections, differences, and trade-offs of existing approaches in terms of (i) safety guarantee and scalability, (ii) safety and reward performance, and (iii) safety after convergence and during training. We also summarize limitations of current methods and discuss potential future directions.
BibTeX
@inproceedings{zhao2023state,
  author = {Weiye Zhao and Tairan He and Rui Chen and Tianhao Wei and Changliu Liu},
  title = {{State-wise Safe Reinforcement Learning: A Survey}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/763},
}
Figure