Abstract
Reinforcement learning (RL) often struggles to maintain safety in systems with high-relative-degree constraints, especially under disturbances and faults. This paper introduces a robust safe RL framework utilizing High-Order Reciprocal Control Barrier Functions (HO-RCBF) and an adaptive safeguarding policy. A key innovation is a gradient manipulation technique that balances performance and safety by projecting conflicting objectives. Simulation results demonstrate that the method provides rigorous safety guarantees and superior performance compared to traditional approaches, ensuring stability even in the presence of actuator faults.
Introduction
In the development of autonomous systems—such as self-driving cars or drones—reinforcement learning is highly effective at optimizing behavior. However, it frequently overlooks safety boundaries during the learning process. Existing "Safe RL" methods face three major hurdles: high-relative-degree constraints where control inputs have a delayed effect on safety boundaries, environmental disturbances and actuator faults that can invalidate safety models, and the persistent trade-off between conservative safety and aggressive performance.
Methodology
The proposed framework employs a multi-layered architecture to address these challenges:
- High-Order Reciprocal Control Barrier Functions (HO-RCBF): Unlike standard barrier functions, Reciprocal CBFs approach infinity at the boundary, providing a stronger deterrent against violations. The "High-Order" extension allows the system to manage constraints where the input-to-state relationship is indirect.
- Gradient Manipulation: To balance performance and safety, the authors define a gradient similarity measure. If the performance goal conflicts with safety, the algorithm projects the performance gradient to ensure it does not violate safety constraints while still making progress toward the objective.
- Adaptive Safeguarding Policy: Instead of a static filter, an adaptive gain ($K_s$) automatically increases when the system is at risk (e.g., due to high disturbances) and decreases when safe to allow for better performance.
- Disturbance/Fault Observer: A nonlinear observer estimates external forces or actuator biases in real-time, allowing the controller to compensate before safety is compromised.
Key Results and Discussion
The framework was evaluated using high-fidelity simulations, including a double integrator mobile robot and an inverted pendulum subject to faults.
Key findings include:
- Robustness to Faults: In inverted pendulum tests, the proposed method maintained safety constraints throughout the learning process, even under unknown time-varying disturbances that caused classical RL and standard safety filters to fail.
- Efficiency through Adaptation: The adaptive HO-RCBF method achieved significantly lower accumulated costs compared to non-adaptive versions. By reducing conservativeness when far from boundaries, the system reached targets faster.
- Theoretical Soundness: The paper provides rigorous Lyapunov-based proofs showing that the composite state (system state, neural network weights, and disturbance estimates) is Uniformly Ultimately Bounded (UUB).
Conclusion
This research demonstrates that decoupling safeguarding from performance optimization allows RL agents to learn complex tasks while maintaining robust safety. By utilizing Reciprocal CBFs and adaptive mechanisms, the system effectively handles the "discrete-time reality" of computer-controlled hardware. Future work will focus on learning system dynamics online to reduce model dependence and exploring computational optimizations for higher-dimensional systems.
Summary updated