Safe Reinforcement Learning Using Advantage-Based Intervention

Nolan Wagener, Byron Boots, Ching-An Cheng

Abstract

Many sequential decision problems involve finding a policy that maximizes total reward while obeying safety constraints. Although much recent research has focused on the development of safe reinforcement learning (RL) algorithms that produce a safe policy after training, ensuring safety during training as well remains an open problem. A fundamental challenge is performing exploration while still satisfying constraints in an unknown Markov decision process (MDP). In this work, we address this problem for the chance-constrained setting. We propose a new algorithm, SAILR, that uses an intervention mechanism based on advantage functions to keep the agent safe throughout training and optimizes the agent's policy using off-the-shelf RL algorithms designed for unconstrained MDPs. Our method comes with strong guarantees on safety during both training and deployment (i.e., after training and without the intervention mechanism) and policy performance compared to the optimal safety-constrained policy. In our experiments, we show that SAILR violates constraints far less during training than standard safe RL and constrained MDP approaches and converges to a well-performing policy that can be deployed safely without intervention.

Introduction

In real-world applications like robotics, reinforcement learning (RL) agents must learn high-performing policies without causing damage during the exploration phase. Standard safe RL frameworks, such as Constrained Markov Decision Processes (CMDPs), typically only guarantee safety for the final policy, often leading to numerous constraint violations during training. Furthermore, CMDP algorithms rely on complex constrained optimization that can be numerically unstable.

While "shields" or intervention mechanisms can prevent unsafe actions, agents often fail to learn safe behavior independently, relying instead on the shield. This paper introduces SAILR (Safe Advantage-based Intervention for Learning policies with Reinforcement), a framework that ensures safety during training, utilizes stable unconstrained RL math, and produces policies that remain safe even after the shield is removed.

Methodology: The SAILR Framework

SAILR bridges the gap between shielding and standard RL by transforming safety interventions into a reward-maximization problem within a surrogate MDP.

Advantage-Based Intervention

During training, SAILR monitors the agent's proposed actions using an intervention rule $\mathcal{G}$. It compares the safety of a proposed action $a$ in state $s$ against a known "backup policy" (e.g., a safe fallback maneuver). This comparison is quantified using an advantage function $\bar{A}(s, a)$. If the agent's action is significantly more dangerous than the backup policy (i.e., $\bar{A}(s, a) > \eta$), the backup policy intervenes and takes over.

The Surrogate MDP

A key innovation of SAILR is the construction of a surrogate MDP, $\widetilde{\mathcal{M}}$. When an intervention occurs, the data collected is modified: from the RL algorithm's perspective, the agent transitions into an absorbing "game over" state $s_\dagger$ and receives a large negative reward. This penalty forces the agent to learn to avoid states and actions that trigger the shield.

Conceptual overview of the intervention mechanism and the construction of the surrogate MDP.

Unconstrained Optimization

By baking safety constraints into the environment's rewards via these artificial penalties, SAILR can utilize any standard, off-the-shelf, unconstrained RL algorithm, such as Proximal Policy Optimization (PPO). This avoids the numerical instabilities associated with solving CMDPs directly.

The SAILR algorithm loop.

Experimental Evaluation

The authors evaluated SAILR on two continuous control robotics tasks:

  • Point Robot: A 2D navigation task requiring a point mass to follow a high-speed circular path while staying within a restricted rectangular safe zone.
  • Half-Cheetah: A modified OpenAI Gym task where the robot must run forward while keeping a specific link within a height range to prevent falling.

Key Results

  • Training Safety: SAILR achieved orders of magnitude fewer safety violations during training compared to CMDP baselines like Constrained Policy Optimization (CPO). In some configurations, it achieved near-zero violations.
  • Deployment Performance: Policies learned under SAILR achieved competitive or superior rewards when deployed without the intervention mechanism. The agents successfully internalized the safety constraints rather than just leaning on the shield.
  • Flexibility: The framework proved effective with various backup policies, including learned neural networks and model-based heuristics.

Discussion and Insights

A core insight of SAILR is that advantage functions are superior to Q-values for shielding. Because advantage measures relative safety, the shield naturally deactivates if the backup policy is ineffective in a safe state, preventing over-conservatism and allowing exploration.

The authors also prove that for this unconstrained reduction to work perfectly, the intervention rule must be "partial." This means that whenever the agent is about to do something unsafe, there must exist some alternative action in its action space that avoids triggering the shield.

While SAILR is robust, its performance is bottlenecked by the intervention rule; a highly conservative shield may block optimal paths. Future work includes using collected data to refine the intervention rule dynamically to reduce this performance bias.

Summary updated

Abstract

Many sequential decision problems involve finding a policy that maximizes total reward while obeying safety constraints. Although much recent research has focused on the development of safe reinforcement learning (RL) algorithms that produce a safe policy after training, ensuring safety during training as well remains an open problem. A fundamental challenge is performing exploration while still satisfying constraints in an unknown Markov decision process (MDP). In this work, we address this problem for the chance-constrained setting. We propose a new algorithm, SAILR, that uses an intervention mechanism based on advantage functions to keep the agent safe throughout training and optimizes the agent's policy using off-the-shelf RL algorithms designed for unconstrained MDPs. Our method comes with strong guarantees on safety during both training and deployment (i.e., after training and without the intervention mechanism) and policy performance compared to the optimal safety-constrained policy. In our experiments, we show that SAILR violates constraints far less during training than standard safe RL and constrained MDP approaches and converges to a well-performing policy that can be deployed safely without intervention. Our code is available at https://github.com/nolanwagener/safe_rl.
BibTeX
@inproceedings{wagener2021safe,
  author = {Nolan Wagener and Byron Boots and Ching-An Cheng},
  title = {{Safe Reinforcement Learning Using Advantage-Based Intervention}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=f73I9SMDf7},
}
Figure