Safe Policy Exploration Improvement via Subgoals

Brian Angulo, Gregory Gorbov, Aleksandr Panov, Konstantin S. YakovlevarXiv 2024 · DOI · arXiv

Introduction

In the field of autonomous navigation, Reinforcement Learning (RL) agents often encounter a fundamental conflict between exploration and safety. To learn effectively, an agent must explore its environment; however, strict safety constraints—such as avoiding collisions—often penalize movement into uncertain or unvisited areas. This frequently leads to "timid" behavior, where the agent remains in a local safe zone and fails to reach distant goals. This challenge is known as the long-horizon problem under safety constraints.

The SPEIS Framework

To address this, the authors propose Safe Policy Exploration Improvement via Subgoals (SPEIS), a hierarchical reinforcement learning framework designed to bridge the gap between safety and effective exploration. SPEIS decomposes the navigation task into two distinct levels:

  1. The Safe Policy (Lower Level): This policy is built on the Soft Actor-Critic (SAC) Lagrangian method. It focuses on maximizing task rewards while ensuring that cumulative safety costs (e.g., obstacle collisions) remain below a predefined threshold. It employs an adaptive safety weight ($\lambda$) to penalize violations dynamically.
  2. The Subgoal Policy (Higher Level): This policy serves as a strategic guide, analyzing the agent's memory of past transitions to generate intermediate "stepping stone" goals.
Conceptual overview of SPEIS: Subgoals bridge the gap between safe exploration and long-distance goals.

The core innovation of SPEIS lies in its integration mechanism. Instead of forcing the agent to follow subgoals strictly, the framework uses Kullback-Leibler (KL) divergence in the loss function. This mathematically nudges the safe policy's action distribution toward the subgoal without overriding primary safety constraints. This allows the agent to explore the environment more purposefully while remaining within the "safe zone."

Experimental Evaluation

The researchers evaluated SPEIS across two rigorous simulated environments:

  • Safety-Gym: A standard benchmark for safe RL, featuring four robot models: Point, Car, Doggo (quadruped), and Sweep (holonomic).
  • POLAMP: A specialized autonomous vehicle simulator utilizing LiDAR data. Tasks ranged from simple single turns (Level 1) to complex narrow corridors requiring multiple precise maneuvers over 20–30 meters (Level 2).

Key Results and Insights

SPEIS consistently outperformed state-of-the-art competitors, including CPO, SAC-Lagrangian, and Lyapunov-RRT, in both safety and mission success.

  • Overcoming Timidity: In the most challenging POLAMP environments (narrow corridors), standard safe RL methods like SAC-Lagrangian failed completely (0% success rate) due to excessive caution. In contrast, SPEIS achieved an 84% success rate.
  • Safety Performance: Across all robotic platforms, SPEIS maintained a remarkably low collision rate of approximately 3%, compared to rates exceeding 20% for standard RL methods in complex tasks.
  • Robustness: Unlike hybrid methods that require perfect environmental maps, SPEIS operates on raw sensor data (LiDAR), making it more applicable to real-world scenarios where full observability is unavailable.

Conclusion and Future Work

SPEIS demonstrates that subgoals are not just planning tools but essential catalysts for safe exploration. By providing near-term targets, the framework prevents agents from becoming trapped in safe-but-static local optima. Future research may explore incorporating model-based approaches to improve early-stage sample efficiency and extending the framework to handle dynamic obstacles like pedestrians.

Summary updated

BibTeX
@misc{angulo2024safe,
  author = {Brian Angulo and Gregory Gorbov and Aleksandr Panov and Konstantin S. Yakovlev},
  title = {{Safe Policy Exploration Improvement via Subgoals}},
  year = {2024},
  eprint = {2408.13881},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.13881},
}
Figure