Safety-Constrained Policy Transfer with Successor Features

Zeyu Feng, Bowen Zhang, Jianxin Bi, Harold Soh

Abstract

In this work, we focus on the problem of safe policy transfer in reinforcement learning: we seek to leverage existing policies when learning a new task with specified constraints. This problem is important for safety-critical applications where interactions are costly and unconstrained policies can lead to undesirable or dangerous outcomes, e.g., with physical robots that interact with humans. We propose a Constrained Markov Decision Process (CMDP) formulation that simultaneously enables the transfer of policies and adherence to safety constraints. Our formulation cleanly separates task goals from safety considerations and permits the specification of a wide variety of constraints. Our approach relies on a novel extension of generalized policy improvement to constrained settings via a Lagrangian formulation. We devise a dual optimization algorithm that estimates the optimal dual variable of a target task, thus enabling safe transfer of policies derived from successor features learned on source tasks. Our experiments in simulated domains show that our approach is effective; it visits unsafe states less frequently and outperforms alternative state-of-the-art methods when taking safety constraints into account.

Introduction

Reinforcement Learning (RL) has demonstrated significant potential across various domains, yet its reliance on trial-and-error exploration poses substantial risks in safety-critical applications. In real-world scenarios, such as human-robot interaction, exploratory failures can lead to catastrophic outcomes. While transfer learning offers a path toward reducing sample complexity by leveraging knowledge from prior "source" tasks, existing frameworks like Successor Features (SF) are typically safety-agnostic, prioritizing reward maximization over constraint satisfaction.

To address this, researchers have introduced Successor Feature Transfer for Constrained Policies (SFT-COP). This framework enables agents to adapt to new tasks while strictly adhering to safety constraints, such as maintaining a low probability of entering hazardous zones. Unlike risk-sensitive methods that merely reduce return variance, SFT-COP explicitly satisfies multi-faceted safety constraints.

Methodology

The SFT-COP framework is built upon three core pillars:

  1. Constrained MDP (CMDP) Formulation: The agent's objective is to maximize a reward function $r$ subject to constraints on a utility function $c$, which represents safety requirements.
  2. Successor Features (SF): By decoupling environment dynamics from specific reward and constraint weights, SFs allow for the instantaneous re-evaluation of source policies under new target parameters.
  3. Lagrangian Dual Optimization: The method employs a Lagrangian formulation to manage constraints, introducing a dual variable ($\lambda$) that acts as a dial to balance reward and safety.

A key theoretical insight of this work is the proof that the performance gap in safe transfer is primarily driven by the error in estimating the optimal Lagrange multiplier $\lambda$ for the target task. The proposed dual optimization algorithm finds the optimal $\lambda$ by treating the policy space as a convex combination of source policies, allowing the agent to synthesize safe behaviors from past experiences.

Experimental Evaluation

The authors evaluated SFT-COP across three distinct environments: a discrete Four-Room Gridworld, a continuous Reacher robotic arm simulation, and SafetyGym, a high-dimensional navigation task involving hazard avoidance.

The SafetyGym environment provides a complex, high-dimensional navigation task for testing safety constraints.

Key Results

SFT-COP demonstrated superior safety and performance across all benchmarks. In the Four-Room domain, it visited unsafe states significantly less frequently than safety-agnostic (SFQL) and risk-sensitive (RaSFQL) baselines. Furthermore, SFT-COP achieved higher rewards faster than "from-scratch" safe RL methods like Constrained Policy Optimization (CPO), which often struggled to converge within the same timeframe.

Performance comparison in the Four-Room Gridworld. SFT-CoP (blue) achieves higher probabilistic rewards than safety-agnostic and risk-sensitive baselines while maintaining safety.

The agent also showed remarkable robustness to uncertainty. In scenarios with probabilistic traps, SFT-COP maintained high safety levels without becoming overly conservative—a common failure mode for risk-sensitive methods that avoid rewards associated with any variance (the "probabilistic reward" problem).

Conclusion and Future Work

SFT-COP provides a robust mechanism for safe policy transfer by effectively balancing reward and safety through dual optimization. While the current implementation utilizes discretized action spaces, future work will explore extensions to fully continuous action spaces using algorithms like Soft Actor-Critic. Additionally, the researchers aim to investigate "zero-shot" safety to eliminate unsafe exploration even during the initial source task training phases.

Summary updated

Abstract

In this work, we focus on the problem of safe policy transfer in reinforcement learning: we seek to leverage existing policies when learning a new task with specified constraints. This problem is important for safety-critical applications where interactions are costly and unconstrained policies can lead to undesirable or dangerous outcomes, e.g., with physical robots that interact with humans. We propose a Constrained Markov Decision Process (CMDP) formulation that simultaneously enables the transfer of policies and adherence to safety constraints. Our formulation cleanly separates task goals from safety considerations and permits the specification of a wide variety of constraints. Our approach relies on a novel extension of generalized policy improvement to constrained settings via a Lagrangian formulation. We devise a dual optimization algorithm that estimates the optimal dual variable of a target task, thus enabling safe transfer of policies derived from successor features learned on source tasks. Our experiments in simulated domains show that our approach is effective; it visits unsafe states less frequently and outperforms alternative state-of-the-art methods when taking safety constraints into account.
BibTeX
@inproceedings{feng2023safety,
  author = {Zeyu Feng and Bowen Zhang and Jianxin Bi and Harold Soh},
  title = {{Safety-Constrained Policy Transfer with Successor Features}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=EELBLO6kgR},
}
Figure