Revisiting Safe Exploration in Safe Reinforcement learning

David Eckel, Baohe Zhang, Joschka BödeckerarXiv 2024 · OpenReview

TL;DR

This paper introduces a new metric, Expected Maximum Consecutive Cost steps (EMCC), to better evaluate safe exploration in reinforcement learning. EMCC addresses the limitation of existing metrics by distinguishing between the severity of unsafe actions based on their consecutive occurrences, rather than just their total count. The authors also propose a new benchmark environment, Circle2D, to facilitate faster evaluation of safe exploration algorithms.

What Problem Does This Paper Address?

Reinforcement Learning (RL) agents learn by trial and error, which can be problematic in safety-critical applications. Safe Reinforcement Learning (SafeRL) aims to ensure agents learn policies that adhere to safety constraints, typically by limiting the expected cumulative cost of their actions. However, existing metrics for evaluating SafeRL often treat all cost violations equally, failing to distinguish between a few severe incidents and many minor ones. This can lead to agents that appear safe based on average costs but might still exhibit risky exploration behaviors, potentially overlooking critical safety nuances. The paper addresses this gap by proposing a metric that captures the severity of unsafe exploration, not just its frequency.

Methods

The core contribution is the Expected Maximum Consecutive Cost steps (EMCC) metric. EMCC is calculated by:

  1. Per Trajectory Calculation: For each trajectory within a training phase, determine the maximum number of consecutive unsafe steps (steps incurring a cost). This is then normalized by the trajectory's length.
  2. Rollout Aggregation: For a given training phase (e.g., the first third, middle third, or last third of training), the maximum of these normalized consecutive cost step counts across all rollouts is taken.
  3. Averaging: EMCC is the average of these values over the specified training phase.

The authors also introduce Circle2D, a new benchmark environment designed for rapid evaluation of safe exploration. It features a global optimum and a circular cost region that agents must avoid. Circle2D offers four levels of difficulty, allowing for progressive challenges in navigating these unsafe zones.

Datasets

  • Circle2D: A newly introduced benchmark environment with four difficulty levels. It's designed to be lightweight and visually intuitive for evaluating safe exploration.
  • Safety-Gymnasium: The paper also uses existing Safety-Gymnasium tasks, specifically SafetyPointCircle1-v0, SafetyPointGoall-v0, and SafetyHopperVelocity-v1, for benchmarking.

Key Insights and Ideas

  • Severity Matters: The paper strongly argues that simply looking at average costs or total cost rates is insufficient for evaluating safe exploration. The duration and consecutiveness of unsafe steps are crucial indicators of an agent's true safety during learning.
  • Exploration vs. Safety Trade-off: The paper highlights the inherent tension between the need for exploration (to find optimal policies) and maintaining safety. EMCC helps to quantify how well algorithms balance this trade-off.
  • On-policy vs. Off-policy Behavior: The study observes distinct trends in EMCC for on-policy and off-policy algorithms. On-policy methods generally show decreasing EMCC over training, indicating safer exploration as training progresses. Off-policy methods, however, can sometimes show increasing EMCC, suggesting persistent exploration of risky areas.
  • Early Termination Challenges: In environments with early termination (like SafetyHopperVelocity-v1), standard metrics can be misleading. EMCC, by normalizing consecutive cost steps by trajectory length, provides a more robust measure of safety even with variable episode lengths.

Key Results

  • EMCC Differentiates: The paper demonstrates that EMCC can distinguish between algorithms that might appear similar using traditional metrics. For instance, in Circle2D-3, CPO and WCSAC had identical cost rates but significantly different EMCC values, suggesting CPO was safer during exploration.
  • On-policy Algorithms Show Improvement: On-policy algorithms like TRPO-Lag consistently showed decreasing EMCC values across various Circle2D levels, indicating they learned to explore more safely over time.
  • Off-policy Algorithms' Mixed Performance: Off-policy algorithms like SAC-Lag and SAC-LB showed more varied EMCC trends. While SAC-Lag sometimes had lower EMCC, indicating safer exploration, it also sometimes had higher cost returns, suggesting a potential trade-off. WCSAC, which incorporates CVaR, was noted to be conservative but not always the safest in terms of exploration.
  • Circle2D Effectiveness: The Circle2D environment proved useful for quickly evaluating algorithms, with some algorithms converging to safe policies while others struggled, especially at higher difficulty levels.

Limitations and Future Work

  • Limitations of Existing Metrics: The paper's primary motivation is the inadequacy of existing metrics (average return, cost rate, CVaR) for capturing the nuances of safe exploration.
  • Future Work: The authors suggest developing new objective functions that directly incorporate the EMCC metric to encourage agents to explore more safely and effectively. They also aim to further leverage the Circle2D environment for algorithm design and evaluation.

Summary updated

Abstract

Safe reinforcement learning (SafeRL) extends standard reinforcement learning with the idea of safety, where safety is typically defined through the constraint of the expected cost return of a trajectory being below a set limit. However, this metric fails to distinguish how costs accrue, treating infrequent severe cost events as equal to frequent mild ones, which can lead to riskier behaviors and result in unsafe exploration. We introduce a new metric, expected maximum consecutive cost steps (EMCC), which addresses safety during training by assessing the severity of unsafe steps based on their consecutive occurrence. This metric is particularly effective for distinguishing between prolonged and occasional safety violations. We apply EMMC in both on- and off-policy algorithm for benchmarking their safe exploration capability. Finally, we validate our metric through a set of benchmarks and propose a new lightweight benchmark task, which allows fast evaluation for algorithm design.
BibTeX
@misc{eckel2024revisiting,
  author = {David Eckel and Baohe Zhang and Joschka Bödecker},
  title = {{Revisiting Safe Exploration in Safe Reinforcement learning}},
  year = {2024},
  eprint = {2409.01245},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.01245},
}
Figure