Abstract
Despite the tremendous success of Reinforcement Learning (RL) algorithms in simulation environments, applying RL to real-world applications still faces many challenges. A major concern is safety, or constraint satisfaction. State-wise constraints are among the most common and challenging constraints in real-world applications, such as autonomous driving and robot manipulation. This paper provides a comprehensive review of existing approaches that address state-wise constraints in RL. Under the framework of State-wise Constrained Markov Decision Process (SCMDP), we discuss the connections, differences, and trade-offs of existing approaches in terms of (i) safety guarantee and scalability, (ii) safety and reward performance, and (iii) safety after convergence and during training. We also summarize limitations of current methods and discuss potential future directions.
Introduction: The Shift to State-wise Constraints
Standard Reinforcement Learning (RL) typically operates on a "trial and error" basis, where agents learn optimal behaviors by exploring environments and receiving rewards. In safety-critical domains like autonomous driving or surgical robotics, this "crash and learn" approach is unacceptable. While the field of Safe RL has traditionally relied on the Constrained Markov Decision Process (CMDP) framework, CMDPs primarily focus on cumulative constraints—ensuring that the average number of violations over time remains below a threshold.
However, real-world safety often requires instantaneous guarantees. A single collision for a self-driving car is a catastrophic failure, regardless of its long-term average performance. This survey explores the State-wise Constrained MDP (SCMDP) framework, where constraints must be satisfied at every individual time step. The core challenge in SCMDP is enforcing these hard boundaries without making the agent so conservative that it fails to perform its primary task.
A Taxonomy of State-wise Safe RL
The survey categorizes the current landscape of research based on two primary dimensions: temporal safety and agent architecture.
1. Temporal Safety: When is Safety Guaranteed?
- Asymptotic Safety (After Convergence): These methods focus on ensuring the agent eventually learns a safe policy. They often utilize soft penalties or "safe critics" to guide the optimization. While effective for reaching high-performance steady states, the agent may still violate constraints during the initial training phases.
- In-training Safety: These methods prioritize safety from the very first step of learning. This is typically achieved through "safety filters," "shielding," or "projection layers" that monitor the RL agent's proposed actions and override them if they lead to a potential constraint violation.
2. Architectural Design: Hierarchical vs. End-to-End
- Hierarchical Agents: These systems employ a two-layer structure. An upper-level RL policy proposes actions based on reward maximization, while a lower-level safety monitor (often utilizing Control Barrier Functions or Reachability Analysis) validates and corrects those actions. This modularity allows for "In-training" safety guarantees.
- End-to-End (E2E) Agents: These agents integrate safety directly into the policy optimization objective. While often more computationally efficient once trained, providing formal safety guarantees during the learning phase is significantly more difficult for E2E architectures.
3. Knowledge of Dynamics
The survey further distinguishes methods by their reliance on environmental models:
- White-box: Assumes known physics and differential equations.
- Black-box: Assumes access to a simulator for forward-looking queries.
- Learned Dynamics: The agent must learn the underlying physics from data, representing the current frontier of the field.
Key Insights and Trade-offs
The Safety-Scalability Gap
There is a fundamental trade-off between the strength of a safety guarantee and the complexity of the system it can support. Methods providing rigorous theoretical guarantees (such as Hamilton-Jacobi reachability) often require "white-box" dynamics and face exponential computational costs, making them difficult to scale to high-dimensional systems like humanoid robots.
The Conservativeness Trap
In-training safety methods often suffer from excessive caution. If a safety filter prevents an agent from ever approaching a boundary, the agent may never explore the state space thoroughly enough to find the optimal path, which frequently resides near those very boundaries.
Safety Indices and Energy Functions
A recurring theme in successful SCMDP research is the use of "Safety Indices" or "Energy Functions." These mathematical constructs map the state space to a value where a specific threshold (e.g., zero) represents the safety boundary. By ensuring the system's "energy" remains below this threshold, researchers can provide mathematical proofs of safety.
Current Benchmarks and Results
While this survey does not introduce new datasets, it synthesizes results across standard Safe RL benchmarks, including autonomous driving simulations, robot manipulation tasks, and classic control problems (e.g., quadrotors and inverted pendulums).
A meta-analysis of nearly 20 major algorithms (including CPO, RL-CBF, and RecoveryRL) reveals that hierarchical methods are generally superior for providing "In-training" safety because they can intervene during the agent's early, high-risk exploration phases. Conversely, E2E methods are often more efficient for deployment but lack the same level of rigorous safety during training.
Limitations and Future Directions
The authors identify three critical hurdles for the future of State-wise Safe RL:
- Model Mismatch: Most "guaranteed safe" methods are sensitive to errors in the underlying physics model. If the model is slightly inaccurate, the safety guarantees may fail in the real world.
- Optimality vs. Safety: Safety layers can "distort" reward signals, making it difficult for agents to reach maximum possible performance.
- High-Dimensional Complexity: Rigorous techniques like SOS programming remain computationally prohibitive for complex, multi-jointed robots.
The survey concludes that the most promising path forward lies in continual learning of dynamics, where agents constantly update their internal models to correct for errors and adapt to changing environments while maintaining safety boundaries.
Summary updated