Abstract
Deep Reinforcement Learning (DRL) has demonstrated significant potential in solving complex robot control tasks, yet it often lacks the formal safety guarantees required for real-world deployment. This paper introduces a novel framework that co-learns a Twin Neural Lyapunov Function (TNLF) alongside a DRL policy to bridge the gap between high-level path planning and low-level safe control. By utilizing the learned Lyapunov function to establish runtime Regions of Attraction (RoA), the system provides explicit "safety bubbles" for high-dimensional robots. Experimental results on a 58-DOF quadruped demonstrate that this approach achieves significantly higher success rates and lower safety violation rates compared to traditional constrained RL methods.
Introduction
While DRL is adept at navigating non-linear dynamics, it typically treats safety as a soft constraint, often implemented via penalty terms in a reward function. This approach frequently leads to "reward hacking," where a robot either becomes overly conservative (refusing to move) or ignores safety penalties entirely to maximize goal-oriented rewards. Furthermore, a disconnect often exists between high-level planners, which suggest paths based on geometry, and low-level controllers, which must account for physical dynamics. A path that appears clear on a 2D map may be dynamically infeasible or unsafe for a complex robot to execute.
To address these challenges, this research proposes a method to provide explicit reachability boundaries. By integrating Lyapunov stability theory with neural networks, the framework ensures that the robot remains within a verifiable "safety bubble" throughout its trajectory.
Methodology
The proposed framework consists of three integrated components: co-learning the stability function, path planning, and runtime monitoring.
1. Co-learning the Twin Neural Lyapunov Function (TNLF)
Rather than treating safety as a post-hoc verification step, the authors propose training the control policy and a Lyapunov function simultaneously. The Twin Neural Lyapunov Function consists of a Lyapunov function $V$ (representing the "energy" or distance from a stable state) and a Lyapunov Q-function (predicting the future value of $V$). The policy is optimized to minimize this Lyapunov value, effectively guiding the robot down an "energy gradient" toward its target waypoint. This co-learning process creates a virtuous cycle: the Lyapunov function provides a dense training signal for the policy, while the policy's exploration helps the Lyapunov function better characterize the system's stability landscape.
2. Path Planning and Waypoint Generation
The system utilizes a standard Rapidly-exploring Random Tree (RRT) planner to generate collision-free paths in a 2D environment. These paths serve as a sequence of waypoints for the low-level controller to follow.
3. Runtime Monitoring with Regions of Attraction (RoA)
The core innovation lies in the runtime monitor. The learned Lyapunov function is used to define a Region of Attraction (RoA)—a mathematical zone where the robot is guaranteed to remain stable. The monitor projects these RoAs onto the environment and uses a line search algorithm to select waypoints that ensure the robot's RoA never intersects with hazardous zones. To maintain real-time performance, RoA radii are pre-computed into a lookup table, allowing for safety-certified decision-making in milliseconds.
Experimental Evaluation
The framework was evaluated using the Safety Gym suite across various robot models and difficulty levels:
- Point: A simple 2D robot.
- Car: A two-wheeled kinematic model.
- Quadruped: A high-dimensional robot with 58 degrees of freedom (DOF).
Testing environments ranged from simple layouts to "Level 3" complexity, featuring up to 128 hazardous zones in a $16 \times 16$ map. The proposed "H-Lyapunov" method was benchmarked against End-to-End (E2E) DRL and Constrained Policy Optimization (CPO).
Results and Discussion
The H-Lyapunov method demonstrated superior performance in both safety and reliability:
- Safety Guarantees: In the most challenging Level-3 tasks, the H-Lyapunov method maintained a safety violation rate between 0.02 and 0.08. In contrast, baseline methods exhibited violation rates as high as 0.33 to 0.88, failing to provide consistent safety in cluttered environments.
- Success Rates: The method achieved near-perfect reach rates (typically 92-100%) across all robot types. Baselines, particularly the high-dimensional quadruped, often failed to reach the goal entirely when faced with complex obstacle configurations.
- The "Safety Tax": The researchers observed that the H-Lyapunov controller often took more steps to reach the goal than unconstrained controllers. This is a deliberate result of the runtime monitor, which forces the robot to slow down or take wider turns to ensure its RoA remains clear of hazards—prioritizing certified safety over raw speed.
Conclusion
This work demonstrates that Neural Lyapunov functions can scale to high-dimensional systems like 58-DOF quadrupeds, providing a bridge between formal stability theory and modern DRL. By moving from implicit penalties to explicit Regions of Attraction, the framework offers a robust solution for safe navigation in complex, cluttered environments. Future work will focus on integrating safety constraints directly into the $RRT^*$ planning phase and adapting the runtime monitor to handle dynamic obstacles and varying terrains.
Summary updated