A Review of Safe Reinforcement Learning: Methods, Theories, and Applications

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Alois Knoll

TL;DR

This paper provides a comprehensive review of Safe Reinforcement Learning (Safe RL), a critical area for deploying RL in real-world applications. It identifies five key challenges ("2H3W") and systematically analyzes existing methods, theories, and applications to guide future research and development in this crucial field.

What Problem Does This Paper Address?

Reinforcement Learning (RL) has shown remarkable success in various domains, but its application in real-world scenarios is hampered by safety concerns. Unacceptable consequences can arise if RL agents operate without safety guarantees, especially in critical applications like autonomous driving and robotics. While safe control has a long history, the integration of safety into RL is still in its early stages. This paper addresses the fundamental question: "How do we guarantee safety when we deploy RL in real-world applications?" It aims to provide a structured overview of the field, identify key challenges, and offer insights for researchers and practitioners.

Methods

The paper categorizes Safe RL methods based on several perspectives:

  • Objective Optimization: Methods that treat cost as an optimization objective alongside reward.
  • Knowledge Utilization: Approaches that leverage external knowledge to guide RL exploration and ensure safety.
  • Policy vs. Value-Based: Distinguishing between methods that directly optimize policies and those that focus on value functions.
  • Agent Number: Differentiating between single-agent and multi-agent Safe RL settings.
  • Model-Based vs. Model-Free: Analyzing methods that rely on learned environment models versus those that do not.

Specific techniques discussed include:

  • Policy Optimization-Based Approaches: Methods like CPO, PCPO, and TRPO adapted for safety constraints.
  • Control Theory-Based Approaches: Utilizing Lyapunov functions, Model Predictive Control (MPC), and Control Barrier Functions (CBFs) for safety guarantees.
  • Formal Methods-Based Approaches: Employing formal verification techniques to ensure safety without probabilistic guarantees.
  • Gaussian Processes-Based Approaches: Leveraging GPs to estimate uncertainties and identify unsafe areas for safer exploration.

Datasets

The paper discusses various benchmarks and environments used for evaluating Safe RL algorithms, including:

  • AI Safety Gridworlds: A 2D grid-based environment for evaluating navigation and task completion with safety constraints.
  • Safety Gym: An OpenAI Gym-based environment with 2D robots (Point, Car, Doggo) that must avoid dangerous areas and objects.
  • Safe Control Gym: An extension of OpenAI Gym integrating traditional control, learning-based control, and RL methods for tasks like cart-pole and quadrotor control.
  • Safe Multi-Agent MuJoCo (Safe MAMuJoCo): An extension of MuJoCo for multi-agent robotics, incorporating obstacles and requiring agents to avoid collisions.
  • Safe Multi-Agent Robosuite (Safe MARobosuite): Based on Robosuite, this benchmark focuses on multi-agent robotic arm manipulation with safety constraints.
  • Safe Multi-Agent Isaac Gym (Safe MAIG): A high-performance, GPU-accelerated environment for multi-agent RL with safety constraints.

Key Insights and Ideas

  • The "2H3W" Framework: The paper proposes a crucial framework of five key problems (Safety Policy, Safety Complexity, Safety Applications, Safety Benchmarks, Safety Challenges) that define the landscape of Safe RL research. This provides a structured way to understand and address the core issues.
  • Safety is Paramount: In many real-world applications, an agent's safety is more critical than its reward, necessitating dedicated safety mechanisms beyond standard RL.
  • Trade-offs are Inevitable: Safe RL inherently involves balancing reward maximization with safety guarantees, which often leads to complex trade-offs that need careful consideration in algorithm design and evaluation.
  • Model-Based vs. Model-Free Trade-offs: Model-based methods often offer better learning efficiency but require accurate models, while model-free methods are more broadly applicable but can be less sample-efficient.
  • The Importance of Benchmarks: The development of specialized Safe RL benchmarks is crucial for fair comparison of algorithms and for bridging the gap between simulation and real-world deployment.

Key Results

The paper does not present novel experimental results but rather synthesizes existing research. Its key contribution lies in its comprehensive review and analysis. The paper highlights:

  • The progress in various Safe RL methods, including policy optimization, control theory, formal methods, and Gaussian processes.
  • The analysis of sample complexity and convergence for different Safe RL algorithms.
  • The application of Safe RL in domains like autonomous driving and robotics.
  • The identification of significant challenges for future Safe RL research, such as human-compatible safety, industrial deployment standards, and safety guarantees in large-scale multi-agent systems.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Human-Compatible Safe RL: Challenges remain in accurately capturing human preferences, dealing with malicious user input, and handling ethical considerations and social dilemmas in multi-agent systems.
  • Industrial Deployment Standards: There is a lack of standardized technical and legal frameworks for deploying Safe RL in industrial applications. Tailored standards for specific applications are needed.
  • Safety Guarantee in Large-Scale MARL: Efficiently guaranteeing safety in environments with a vast number of agents is a significant challenge.
  • Theory of Safe MARL: Further theoretical work is needed on credit assignment, non-stability, scalability, and handling multiple heterogeneous constraints in MARL.
  • Robust MARL: Addressing mismatches between nominal and real systems, and ensuring zero cost without extensive parameter tuning, are open problems.
  • Trade-off Balances: Effectively managing the trade-offs between exploration/exploitation and reward/cost is crucial, especially in competitive and mixed-game settings.
  • Future Directions: The paper suggests exploring Safe MARL with game theory, information theory, and human-brain/biology inspiration. Learning safe and diverse behaviors from human feedback and improving human-robot interactions are also highlighted as key areas for future research.

Summary updated

Abstract

Reinforcement Learning (RL) has achieved tremendous success in many complex decision-making tasks. However, safety concerns are raised during deploying RL in real-world applications, leading to a growing demand for safe RL algorithms, such as in autonomous driving and robotics scenarios. While safe control has a long history, the study of safe RL algorithms is still in the early stages. To establish a good foundation for future safe RL research, in this paper, we provide a review of safe RL from the perspectives of methods, theories, and applications. First, we review the progress of safe RL from five dimensions and come up with five crucial problems for safe RL being deployed in real-world applications, coined as “2H3W” . Second, we analyze the algorithm and theory progress from the perspectives of answering the “2H3W” problems. Particularly, the sample complexity of safe RL algorithms is reviewed and discussed, followed by an introduction to the applications and benchmarks of safe RL algorithms. Finally, we open the discussion of the challenging problems in safe RL, hoping to inspire future research on this thread. To advance the study of safe RL algorithms, we release an open-sourced repository containing major safe RL algorithms at the link.
BibTeX
@article{gu2024review,
  author = {Shangding Gu and Long Yang and Yali Du and Guang Chen and Florian Walter and Jun Wang and Alois Knoll},
  title = {{A Review of Safe Reinforcement Learning: Methods, Theories, and Applications}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3457538},
}
Figure