Model-free Neural Lyapunov Control for Safe Robot Navigation

Zikang Xiong, Joe Eappen, Ahmed H. Qureshi, Suresh Jagannathan

Abstract

Deep Reinforcement Learning (DRL) has demonstrated significant potential in solving complex robot control tasks, yet it often lacks the formal safety guarantees required for real-world deployment. This paper introduces a novel framework that co-learns a Twin Neural Lyapunov Function (TNLF) alongside a DRL policy to bridge the gap between high-level path planning and low-level safe control. By utilizing the learned Lyapunov function to establish runtime Regions of Attraction (RoA), the system provides explicit "safety bubbles" for high-dimensional robots. Experimental results on a 58-DOF quadruped demonstrate that this approach achieves significantly higher success rates and lower safety violation rates compared to traditional constrained RL methods.

Introduction

While DRL is adept at navigating non-linear dynamics, it typically treats safety as a soft constraint, often implemented via penalty terms in a reward function. This approach frequently leads to "reward hacking," where a robot either becomes overly conservative (refusing to move) or ignores safety penalties entirely to maximize goal-oriented rewards. Furthermore, a disconnect often exists between high-level planners, which suggest paths based on geometry, and low-level controllers, which must account for physical dynamics. A path that appears clear on a 2D map may be dynamically infeasible or unsafe for a complex robot to execute.

To address these challenges, this research proposes a method to provide explicit reachability boundaries. By integrating Lyapunov stability theory with neural networks, the framework ensures that the robot remains within a verifiable "safety bubble" throughout its trajectory.

Methodology

The proposed framework consists of three integrated components: co-learning the stability function, path planning, and runtime monitoring.

1. Co-learning the Twin Neural Lyapunov Function (TNLF)

Rather than treating safety as a post-hoc verification step, the authors propose training the control policy and a Lyapunov function simultaneously. The Twin Neural Lyapunov Function consists of a Lyapunov function $V$ (representing the "energy" or distance from a stable state) and a Lyapunov Q-function (predicting the future value of $V$). The policy is optimized to minimize this Lyapunov value, effectively guiding the robot down an "energy gradient" toward its target waypoint. This co-learning process creates a virtuous cycle: the Lyapunov function provides a dense training signal for the policy, while the policy's exploration helps the Lyapunov function better characterize the system's stability landscape.

2. Path Planning and Waypoint Generation

The system utilizes a standard Rapidly-exploring Random Tree (RRT) planner to generate collision-free paths in a 2D environment. These paths serve as a sequence of waypoints for the low-level controller to follow.

3. Runtime Monitoring with Regions of Attraction (RoA)

The core innovation lies in the runtime monitor. The learned Lyapunov function is used to define a Region of Attraction (RoA)—a mathematical zone where the robot is guaranteed to remain stable. The monitor projects these RoAs onto the environment and uses a line search algorithm to select waypoints that ensure the robot's RoA never intersects with hazardous zones. To maintain real-time performance, RoA radii are pre-computed into a lookup table, allowing for safety-certified decision-making in milliseconds.

Experimental Evaluation

The framework was evaluated using the Safety Gym suite across various robot models and difficulty levels:

  • Point: A simple 2D robot.
  • Car: A two-wheeled kinematic model.
  • Quadruped: A high-dimensional robot with 58 degrees of freedom (DOF).

Testing environments ranged from simple layouts to "Level 3" complexity, featuring up to 128 hazardous zones in a $16 \times 16$ map. The proposed "H-Lyapunov" method was benchmarked against End-to-End (E2E) DRL and Constrained Policy Optimization (CPO).

Results and Discussion

The H-Lyapunov method demonstrated superior performance in both safety and reliability:

  • Safety Guarantees: In the most challenging Level-3 tasks, the H-Lyapunov method maintained a safety violation rate between 0.02 and 0.08. In contrast, baseline methods exhibited violation rates as high as 0.33 to 0.88, failing to provide consistent safety in cluttered environments.
  • Success Rates: The method achieved near-perfect reach rates (typically 92-100%) across all robot types. Baselines, particularly the high-dimensional quadruped, often failed to reach the goal entirely when faced with complex obstacle configurations.
  • The "Safety Tax": The researchers observed that the H-Lyapunov controller often took more steps to reach the goal than unconstrained controllers. This is a deliberate result of the runtime monitor, which forces the robot to slow down or take wider turns to ensure its RoA remains clear of hazards—prioritizing certified safety over raw speed.

Conclusion

This work demonstrates that Neural Lyapunov functions can scale to high-dimensional systems like 58-DOF quadrupeds, providing a bridge between formal stability theory and modern DRL. By moving from implicit penalties to explicit Regions of Attraction, the framework offers a robust solution for safe navigation in complex, cluttered environments. Future work will focus on integrating safety constraints directly into the $RRT^*$ planning phase and adapting the runtime monitor to handle dynamic obstacles and varying terrains.

Summary updated

Abstract

Model-free Deep Reinforcement Learning (DRL) controllers have demonstrated promising results on various challenging non-linear control tasks. While a model-free DRL algorithm can solve unknown dynamics and high-dimensional problems, it lacks safety assurance. Although safety constraints can be encoded as part of a reward function, there still exists a large gap between an RL controller trained with this modified reward and a safe controller. In contrast, instead of implicitly encoding safety constraints with rewards, we explicitly colearn a Twin Neural Lyapunov Function (TNLF) with the control policy in the DRL training loop and use the learned TNLF to build a runtime monitor. Combined with the path generated from a planner, the monitor chooses appropriate waypoints that guide the learned controller to provide collision-free control trajectories. Our approach inherits the scalability advantages from DRL while enhancing safety guarantees. Our experimental evaluation demonstrates the effectiveness of our approach compared to DRL with augmented rewards and constrained DRL methods over a range of high-dimensional safety-sensitive navigation tasks.
BibTeX
@inproceedings{xiong2022model,
  author = {Zikang Xiong and Joe Eappen and Ahmed H. Qureshi and Suresh Jagannathan},
  title = {{Model-free Neural Lyapunov Control for Safe Robot Navigation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=AlL4U82lRbg},
}
Figure