Adversarial Behavior Exclusion for Safe Reinforcement Learning

Md Asifur Rahman, Tongtong Liu, Sarra M. AlqahtaniIJCAI 2023 · DOI

TL;DR

This paper introduces AdvEx-RL, a novel framework for safe reinforcement learning (RL) that learns to exclude unsafe behaviors rather than just constrain them. By training an "adversary" to identify unsafe actions and then learning a separate "safety policy" to avoid them, AdvEx-RL achieves robust safety performance even under perturbations, acting as a task-agnostic firewall.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents learn through exploration, which inherently involves trying out actions that might be unsafe. This makes deploying RL in real-world applications like robotics or autonomous driving challenging, where safety is paramount. Existing approaches to safe RL often try to optimize task performance and safety simultaneously, which can lead to suboptimal policies, or impose explicit constraints that are brittle and struggle with unknown or changing environments. Furthermore, these methods are often vulnerable to adversarial attacks that can exploit their weaknesses, leading to unsafe behavior. This paper addresses the critical need for RL methods that are not only safe but also robust to perturbations and environmental changes, without sacrificing task performance.

Methods

AdvEx-RL employs a two-phase approach:

  1. Learning to be Unsafe (Adversary Training): An "adversary" RL agent is trained to maximize safety violations. This adversary learns an optimal policy ($\pi_{adv}$) that identifies and exploits unsafe behaviors within the environment. The objective here is to learn a robust estimate of safety violations, represented by the optimal state-action value function ($Q_{adv}^*$). This phase uses an off-policy learning method with entropy regularization to encourage exploration of unsafe states.
  2. Learning to be Safe from the Worst Behavior (Safety Policy Training): A separate, task-agnostic "safety policy" ($\pi_{safe}$) is trained. This policy aims to maximize its divergence (KL divergence) from the learned adversary policy ($\pi_{adv}$). The intuition is that by behaving as differently as possible from the adversary (which seeks to violate safety), the safety policy will naturally avoid unsafe actions. This is further refined by optimizing for state-pairwise safety, ensuring the agent can transition to safe states.

During online execution, AdvEx-RL uses a "post-posed shield." This shield monitors the agent's actions. If the task policy selects an action that the $Q_{adv}^*$ function predicts will lead to a safety violation (exceeding a threshold $T_{safety}$), the shield intervenes and replaces the action with one chosen by the safety policy. This creates a safety firewall that can be integrated with any existing RL task policy.

Datasets

The paper does not introduce new datasets. Experiments are conducted on:

  • Three continuous MuJoCo CMDP environments: Maze, Navigation 1, and Navigation 2.
  • SafetyGym environments.

Key Insights and Ideas

  • Safety as Exclusion, Not Just Constraint: The core idea is to learn what not to do by identifying and actively avoiding unsafe behaviors, rather than just trying to satisfy safety constraints. This shifts the paradigm from "how to be safe" to "how to be unsafe, and then avoid that."
  • Adversarial Learning for Safety: Using an adversary to discover unsafe behaviors is a powerful way to generate a comprehensive understanding of potential risks, even in complex environments.
  • Task-Agnostic Safety Firewall: The safety policy and shield are designed to be independent of the task policy, allowing them to be plugged into any RL system without re-training the entire agent. This modularity is a significant practical advantage.
  • Robustness through Divergence: Maximizing the KL divergence between the safety policy and the adversary policy is a theoretically grounded way to ensure the safety policy avoids the adversary's unsafe strategies.
  • Interpretability via Safety-CAPS: The paper extends an existing explainability method (CAPS) to "safety-CAPS," which visualizes the impact of safety violations and the safety policy's interventions, making the agent's behavior more understandable to humans.

Key Results

  • Robustness: AdvEx-RL significantly outperforms 7 baselines in safety performance (over 75% safety) and success-safety (over 80%) under various levels of external action space perturbations and changes in environment dynamics. This robustness is a key differentiator.
  • Performance Trade-off: While safety is prioritized, AdvEx-RL demonstrates a good balance between safety and task performance, achieving high success-safety rates.
  • Interpretability: A user study with 41 participants showed that safety-CAPS graphs generated by AdvEx-RL effectively communicate the agent's behavior and the impact of the safety policy, with users demonstrating good understanding.
  • No Deadlocks Observed: In their deadlock analysis, the authors found no deadlock cycles in baselines or AdvEx-RL across tested environments, even under AAA perturbations.

Limitations and Future Work

  • Deadlock Potential: While not observed in their experiments, the authors acknowledge that providing safety in separate policies can be prone to deadlocks, where the agent might loop between safe and unsafe states. They suggest that deadlocks, while potentially hindering task completion, can sometimes act as a safe fallback.
  • Computational Cost: Training an adversary and a separate safety policy adds computational overhead compared to standard RL.
  • Future Work: The authors plan to extend the framework to multi-agent settings and integrate it with more advanced explainable AI methods. They also aim to explore how to further improve the balance between task performance and safety.

Summary updated

BibTeX
@inproceedings{rahman2023adversarial,
  author = {Md Asifur Rahman and Tongtong Liu and Sarra M. Alqahtani},
  title = {{Adversarial Behavior Exclusion for Safe Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/54},
}
Figure