Researchers at Seoul National University have introduced Safe CoR, a novel framework designed to bridge the critical gap between high-performance autonomous navigation and the stringent safety requirements necessary for real-world deployment. By leveraging a dual-expert imitation learning approach, Safe CoR addresses the inherent trade-offs in Safe Reinforcement Learning (Safe RL).
Abstract
Safe Reinforcement Learning (Safe RL) often faces a conflict between maximizing rewards and adhering to safety constraints, frequently resulting in either unsafe behavior or overly conservative performance. This paper proposes Safe CoR, a framework that utilizes two distinct types of expert demonstrations: a "Reward Expert" focused on performance and a "Safe Expert" focused on hazard avoidance. By integrating a relative distance metric called Constraint Reward (CoR) into both the agent's objective and its constraints, the framework achieves a dynamic balance between speed and safety. Evaluated on Safety Gym, MetaDrive, and real-world robotic hardware, Safe CoR demonstrated up to a 39% performance boost and an 88% reduction in safety violations.
Introduction: The Safety-Performance Trade-off
Reinforcement Learning (RL) enables agents to adapt to complex environments through trial and error. However, in domains like autonomous driving, failures during the learning phase can be catastrophic. Safe RL attempts to mitigate this by incorporating mathematical constraints into the optimization process.
Existing methods often suffer from a "Goldilocks" problem: constraints that are too loose lead to frequent crashes during training, while constraints that are too strict result in the "frozen robot" problem, where the agent becomes too cautious to reach its goal. Safe CoR addresses this by using expert data not merely to copy behavior, but to provide a dynamic safety margin that adjusts based on the agent's proximity to risk.
Methodology: The Dual-Expert Framework
The core innovation of Safe CoR lies in its Dual-Expert approach combined with the Constraint Reward (CoR) metric.
- Dual Experts: Unlike traditional Imitation Learning that relies on a single optimal expert, Safe CoR utilizes a Reward Expert (optimized for goal reaching) and a Safe Expert (optimized for hazard avoidance).
- Constraint Reward (CoR): This mathematical term measures the agent's relative distance to these two experts. A high CoR indicates the agent is following the performance-oriented path, while a low CoR suggests a shift toward the safety-oriented path.
- Optimization Integration: Safe CoR integrates the CoR term into two distinct components of the RL algorithm:
- The Objective: It serves as a guide to help the agent maximize task-related rewards.
- The Constraint: It acts as a "safety regularizer." In risky scenarios, the CoR term effectively tightens the safety constraints, forcing the agent to prioritize the Safe Expert's behavior.
Experimental Evaluation
The researchers evaluated Safe CoR across three diverse environments to test its robustness and generalizability, ranging from standard benchmarks to real-world hardware.
- Safety Gym: A benchmark for safe RL involving robots navigating around hazards.
- MetaDrive: A high-fidelity autonomous driving simulator featuring LiDAR feedback and complex traffic scenarios.
- Jackal Platform: A physical ground robot used to validate the framework's "sim-to-real" performance.
Key Results and Discussion
Integrating Safe CoR into state-of-the-art Safe RL algorithms (such as SDAC, CPO, and OffTRC) yielded significant improvements. In the MetaDrive simulator, the framework achieved a 29.6% increase in driving score alongside a 61.9% reduction in violations.
On the real-world Jackal robot, the framework reduced constraint violations by 88% compared to standard Safe RL. A critical finding from the ablation studies was that applying CoR to both the reward and the constraint functions is essential; omitting either led to a failure to balance performance and safety effectively.
Conclusion
Safe CoR demonstrates that safety in autonomous systems should be treated as a spectrum rather than a binary state. By navigating between aggressive and cautious behavioral styles, the framework provides a dynamic governor that ensures safety without sacrificing performance. Future work will explore the application of this dual-expert approach to multi-agent environments and the erratic nature of human-driven traffic.
Summary updated