Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret

Haitham Bou-Ammar, Rasul Tutunov, Eric Eaton

Introduction

Reinforcement learning (RL) has shown immense promise in solving complex control tasks, yet its application to real-world physical systems remains limited by data inefficiency and safety concerns. Standard RL agents typically learn each new task from scratch, a process known as tabula rasa learning, which is prohibitively expensive in high-dimensional robotics. Lifelong RL aims to mitigate this by transferring knowledge across a sequence of tasks. However, existing lifelong methods often suffer from linear regret—meaning their performance gap relative to an optimal baseline does not vanish—and lack formal safety guarantees, risking catastrophic failure during exploration.

This paper introduces a novel safe lifelong policy gradient RL algorithm. By factoring policy parameters into a shared latent basis and task-specific coefficients, the proposed method enables rapid knowledge transfer while strictly enforcing safety constraints and achieving guaranteed sublinear regret.

Problem Statement

The authors identify two critical gaps in current lifelong RL research:

  1. Non-vanishing Regret: Existing methods exhibit linear regret $\mathcal{O}(R)$, where $R$ is the number of learning rounds. This implies that the agent does not become significantly more efficient as it encounters more tasks.
  2. Safety Vulnerabilities: Transfer methods typically employ unconstrained optimization. In physical environments, unconstrained exploration can lead to unsafe behaviors, such as exceeding joint torque limits or entering restricted spatial zones.

The objective is to develop a lifelong RL framework that ensures rapid adaptation through shared knowledge, mathematically guarantees sublinear regret, and maintains strict adherence to safety constraints.

Methodology

The framework treats lifelong RL as an online regret minimization problem. The core assumption is that policy parameters for any task can be represented as a linear combination of a shared latent basis $\boldsymbol{L}$ (global knowledge) and a task-specific coefficient vector $\boldsymbol{s}$.

To optimize these parameters, the authors utilize a "Follow the Regularized Leader" (FTRL) approach, implemented via a two-step alternating optimization at each round:

  1. Unconstrained Update: The shared basis $\boldsymbol{L}$ and coefficients $\boldsymbol{s}$ are updated using standard policy gradient methods (e.g., eREINFORCE or eNAC) to minimize accumulated loss, initially ignoring constraints.
  2. Constrained Projection: The unconstrained solution is projected back into a predefined "safe" constraint set. To maintain computational tractability, the authors linearize the loss function around the previous round's safe solution. This allows the use of Semi-Definite Programs (SDPs) and Second-Order Cone Programs (SOCPs) to compute the Bregman divergence projection efficiently.

Experimental Evaluation

The algorithm was evaluated on three distinct simulated dynamical systems, with 10 tasks generated for each by varying parameters such as mass, damping, and inertia:

  • Simple Mass-Spring-Damper: A basic control task to reach a target state.
  • Cart-Pole: The classic benchmark of balancing a pole on a moving cart.
  • Quadrotor UAV: A high-dimensional, chaotic task focused on rotational velocity stabilization.
The three simulated dynamical systems used for benchmarking: mass-spring-damper, cart-pole, and quadrotor UAV.

Results and Discussion

The proposed method demonstrates significant improvements in both theoretical guarantees and empirical performance.

Theoretical and Empirical Regret

The authors prove that their algorithm achieves a sublinear regret bound of $\mathcal{O}(\sqrt{R})$ over $R$ rounds. Empirically, the method consistently outperformed both standard Policy Gradients (eNAC) and the state-of-the-art lifelong learner (PG-ELLA) in learning speed and final performance across all domains.

Safety Enforcement

A key highlight is the algorithm's ability to maintain safety during the learning process. In the complex Quadrotor environment, the proposed method reached a safe policy region after only one task observation. In contrast, standard PG and PG-ELLA required over 500 observations to achieve safety.

Policy parameter trajectories for the simple mass system, demonstrating how the proposed safe PG methods remain within the defined 'safe' region compared to unconstrained baselines.

Conclusion and Future Work

This research establishes that sublinear regret and strict safety are achievable in lifelong RL. By linearizing complex policy gradient losses, the authors provide a tractable path for safe knowledge transfer. While currently focused on log-concave policy distributions like Gaussians, future work will involve extending this framework to deep neural network policies and validating the approach on physical robotic hardware.

Summary updated

BibTeX
@inproceedings{bouammar2015safe,
  author = {Haitham Bou-Ammar and Rasul Tutunov and Eric Eaton},
  title = {{Safe Policy Search for Lifelong Reinforcement Learning with Sublinear Regret}},
  booktitle = {International Conference on Machine Learning},
  year = {2015},
  url = {https://openreview.net/forum?id=BkVeOh-dZr},
}
Figure