End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

Richard Cheng, Gábor Orosz, Richard M. Murray, Joel W. Burdick

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in complex decision-making tasks. However, its fundamental reliance on trial-and-error exploration poses significant risks in safety-critical, real-world applications such as autonomous driving or industrial robotics. A single exploratory failure can lead to catastrophic hardware damage. Traditional "Safe RL" approaches often provide only approximate safety guarantees after training or exhibit extreme conservatism that hinders learning efficiency. This paper introduces RL-CBF, a framework designed to ensure provable safety throughout the entire learning process without compromising performance.

The RL-CBF Framework

The RL-CBF architecture integrates model-free RL with model-based control theory to create a "safety shield" for the agent. The system consists of three primary components:

  1. Model-Free RL: Standard algorithms like TRPO or DDPG serve as the "brain," learning high-performance control policies.
  2. Control Barrier Functions (CBFs): These act as a filter. If the RL agent proposes an action that would lead to a safety violation, the CBF minimally modifies it to the nearest safe action via a Quadratic Program (QP).
  3. Gaussian Processes (GPs): To account for model uncertainty (e.g., unknown friction), GPs model environment dynamics online, ensuring the CBF remains safety-aware even with an imperfect initial model.
Architecture of the RL-CBF framework.

Guiding Control and Policy Projection

A key innovation of this work is the Guiding Control mechanism. In typical shielded RL, the agent is unaware of the modifications made by the safety filter, which can lead to a mismatch between the agent's internal state and the environment. RL-CBF addresses this by training the RL agent on the actual safe actions deployed in the environment. This effectively projects the RL policy onto the set of safe policies, narrowing the search space and guiding the agent toward high-performance, safe regions of the state space.

Policy space visualization of safety projection and guiding control.

Experimental Results

The framework was evaluated on two benchmarks: an Inverted Pendulum with angular constraints and a Simulated Car Following task requiring collision avoidance.

  • Zero Violations: Unlike standard TRPO and DDPG, which frequently crashed during training, RL-CBF maintained 100% safety throughout the learning process.
  • Sample Efficiency: By preventing exploration of unsafe states, RL-CBF converged to optimal policies significantly faster than baselines. In car-following, it successfully learned fuel-efficient behavior while maintaining a hard 2-meter headway constraint.
  • Robustness: The system demonstrated graceful degradation; even when a perfectly safe action was physically impossible, the CBF provided the "least unsafe" action to minimize risk.

Conclusion

RL-CBF represents a significant step toward deploying RL in real-world safety-critical systems. By treating safety as a hard mathematical constraint rather than a reward penalty, the framework ensures zero violations during training while simultaneously improving learning speed. Future work will explore methods to learn the safe set boundaries directly from data, reducing the reliance on pre-defined safety functions.

Summary updated

Abstract

Reinforcement Learning (RL) algorithms have found limited success beyond simulated applications, and one main reason is the absence of safety guarantees during the learning process. Real world systems would realistically fail or break before an optimal controller can be learned. To address this issue, we propose a controller architecture that combines (1) a model-free RL-based controller with (2) model-based controllers utilizing control barrier functions (CBFs) and (3) on-line learning of the unknown system dynamics, in order to ensure safety during learning. Our general framework leverages the success of RL algorithms to learn high-performance controllers, while the CBF-based controllers both guarantee safety and guide the learning process by constraining the set of explorable polices. We utilize Gaussian Processes (GPs) to model the system dynamics and its uncertainties. Our novel controller synthesis algorithm, RL-CBF, guarantees safety with high probability during the learning process, regardless of the RL algorithm used, and demonstrates greater policy exploration efficiency. We test our algorithm on (1) control of an inverted pendulum and (2) autonomous car-following with wireless vehicle-to-vehicle communication, and show that our algorithm attains much greater sample efficiency in learning than other state-of-the-art algorithms and maintains safety during the entire learning process.
BibTeX
@inproceedings{cheng2019end,
  author = {Richard Cheng and Gábor Orosz and Richard M. Murray and Joel W. Burdick},
  title = {{End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33013387},
}
Figure