Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environments

Hongyi Chen, Changliu Liu

Abstract

This study proposes a safe and sample-efficient reinforcement learning (RL) framework to address two major challenges in developing applicable RL algorithms: satisfying safety constraints and efficiently learning with limited samples. To guarantee safety in real-world complex environments, we use the safe set algorithm (SSA) to monitor and modify the nominal controls, and evaluate SSA+RL in a clustered dynamic environment which is challenging to be solved by existing RL algorithms. To improve the learning efficiency, we propose three techniques: (1) adapting the SSA to avoid over-conservatism; (2) encouraging safe exploration using random network distillation with safety constraints; (3) improving policy convergence by treating SSA as expert demonstrations. Experimental results show our framework achieves superior safety and solves tasks with substantially fewer episodes.

Introduction: The Deployment Gap in Safe RL

Reinforcement Learning (RL) often faces a "deployment gap" when moving from simulation to the real world. While an agent in a simulator can afford thousands of collisions to learn, real-world applications like autonomous driving require zero-collision guarantees from the start. Furthermore, real-world data is expensive, making sample efficiency a critical bottleneck, especially in "reward-sparse" environments where feedback is infrequent.

Existing "Safe RL" methods often rely on safety filters that can be overly conservative, leading to inefficient paths or "freezing" behavior in crowded environments. This paper addresses these issues by integrating a control-theoretic "shield" with proactive planning and online expert demonstrations.

Methodology: Enhancing the Safe Set Algorithm

The core of the proposed framework is the Safe Set Algorithm (SSA), which acts as a monitor to modify unsafe control commands. To overcome the limitations of standard SSA, three novel enhancements are introduced:

1. Adapted SSA for Proactive Shielding

Standard SSA is often reactive, focusing only on the immediate next step. The Adapted SSA predicts the future trajectories of nearby obstacles. By calculating a "safest direction" based on the environment's geometry (using eigenvectors), the algorithm biases the optimization process to nudge the robot toward open space before it becomes cornered.

Conceptual comparison between vanilla SSA and the proposed Adapted SSA, showing proactive navigation.

2. Safe Exploration via RND

To prevent the agent from getting stuck in local optima, Random Network Distillation (RND) is used to encourage exploration of novel states. Crucially, these exploratory actions are still filtered through the SSA shield, ensuring that "curiosity" does not compromise safety.

3. Learning from SSA Demonstrations (Online LfD)

Rather than requiring pre-collected human data, the framework treats the SSA itself as an expert. When the SSA corrects an unsafe RL action, the corrected action is stored in a demonstration buffer. The agent then learns directly from these real-time corrections, which significantly accelerates policy convergence.

Experimental Results

The framework was evaluated in a Clustered Dynamic Environment featuring 50 moving obstacles in a reward-sparse 2D arena.

Performance comparison across baseline models, exploration strategies, adapted SSA, and learning from SSA demonstrations.

The LfD+SSA+RL variant demonstrated remarkable improvements:

  • Success Rate: Achieved 89.8%, compared to just 2% for standard TD3 and 50.2% for vanilla SSA+RL.
  • Safety: Collision rates dropped to 0.8%, whereas standard RL suffered a 31.7% collision rate.
  • Sample Efficiency: The model reached its performance threshold in just 26 episodes (approx. 8,500 interactions). This represents a 38x improvement over the baseline SSA+RL, which required 254 episodes (240,000 interactions).

Conclusion

By transforming safety constraints into a teaching mechanism and incorporating proactive obstacle avoidance, this framework bridges the gap between theoretical safety and practical efficiency. While challenges remain regarding perfect perception and inevitable collisions in extreme density, this approach offers a robust path forward for deploying RL in complex, dynamic real-world environments.

Summary updated

Abstract

This study proposes a safe and sample-efficient reinforcement learning (RL) framework to address two major challenges in developing applicable RL algorithms: satisfying safety constraints and efficiently learning with limited samples. To guarantee safety in real-world complex environments, we use the safe set algorithm (SSA) to monitor and modify the nominal controls, and evaluate SSA+RL in a clustered dynamic environment which is challenging to be solved by existing RL algorithms. However, the SSA+RL framework is usually not sample-efficient especially in reward-sparse environments, which has not been addressed in previous safe RL works. To improve the learning efficiency, we propose three techniques: (1) avoiding behaving overly conservative by adapting the SSA; (2) encouraging safe exploration using random network distillation with safety constraints; (3) improving policy convergence by treating SSA as expert demonstrations and directly learn from that. The experimental results show that our framework can achieve better safety performance compare to other safe RL methods during training and solve the task with substantially fewer episodes. Project website: https://hychen-naza.github.io/projects/Safe_RL/.
BibTeX
@inproceedings{chen2022safe,
  author = {Hongyi Chen and Changliu Liu},
  title = {{Safe and Sample-efficient Reinforcement Learning for Clustered Dynamic Environments}},
  booktitle = {IEEE Control. Syst. Lett},
  year = {2022},
  url = {https://openreview.net/forum?id=8yu6Dcrtga},
}
Figure