Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey

Milan Ganai, Sicun Gao, Sylvia L. Herbert

This survey paper provides a comprehensive overview of how researchers are bridging the gap between formal safety verification and modern reinforcement learning (RL). By integrating Hamilton-Jacobi (HJ) reachability analysis into the RL loop, the field is moving toward autonomous systems that are not just high-performing, but provably safe even in complex, high-dimensional environments.

TL;DR

This survey maps the emerging intersection of formal safety verification and Reinforcement Learning, highlighting how "HJ reachability estimation" methods overcome the traditional curse of dimensionality. By reformulating safety as a learnable value function, these approaches enable rigorous safety guarantees for complex robots—like vision-guided racing cars and lidar-equipped drones—that were previously too complex for formal methods.

What Problem Does This Paper Address?

Reinforcement Learning is exceptionally good at solving complex tasks, but it is notoriously "black-box." In safety-critical applications like autonomous driving or robotics, we need more than just high average performance; we need rigorous guarantees that the system won't fail.

Traditional Hamilton-Jacobi (HJ) reachability analysis is the gold standard for these guarantees. It identifies the "reachable set"—the collection of states from which a system might be forced into a failure state despite its best efforts. However, HJ reachability has historically suffered from the curse of dimensionality. Because it relies on grid-based dynamic programming to solve partial differential equations (PDEs), it was practically limited to systems with about 6 dimensions. This survey explores how recent work uses learning-based methods to scale these safety certificates to systems with over 100 dimensions.

Methods

The paper categorizes methods based on how they integrate HJ reachability with learning:

  • Traditional HJ for Learned Controls: These methods use model-based approaches (like Gaussian Processes) to reconstruct system dynamics and solve the HJ equations analytically. They often act as a "safety filter" or "shield" for a separate RL policy.
  • HJ Reachability Estimation (Model-Free): The core innovation in recent years is treating the HJ reachability value function like a standard RL value function. By using recursive Bellman updates and Temporal Difference (TD) learning, researchers can learn safety certificates directly from data without an explicit system model.
  • Discounted Formulations: To ensure these learning algorithms converge, researchers introduce discount factors ($\gamma$). While this slightly alters the "true" safety set, it enables the use of stable RL algorithms like DQN or Soft Actor-Critic.
  • Reach-Avoid Frameworks: These methods combine the goal-seeking objective ("reach") and the safety objective ("avoid") into a single composite value function, allowing for unified optimization.
  • Shielding and RCRL: Frameworks like Reachability Constrained Reinforcement Learning (RCRL) and Reachability Estimation for Safe Policy Optimization (RESPO) provide specific mathematical structures to prioritize safety over rewards during training.

Datasets

The surveyed works evaluate their methods on a variety of benchmarks, ranging from classic control to high-dimensional robotics:

  • Classic RL Environments: Lunar Lander and Jumping Half-Cheetah (18D).
  • Safety Benchmarks: Safety Gym (static hazards), Safe PyBullet, and MuJoCo.
  • High-Dimensional Vision/Lidar Tasks: The Learn-to-Race environment (40D vision-based racing) and lidar-based navigation tasks (up to 112D).
  • Real-World Hardware: Hardware experiments involving quadruped robots (like the Unitree A1 or Mini Cheetah) and small autonomous robot cars.

Key Insights and Ideas

  • The Bellman Bridge: The most significant "aha moment" is the realization that the HJ reachability PDE can be approximated by a discrete-time Bellman backup. This makes safety verification compatible with the entire toolbox of modern deep RL.
  • Scaling via Approximation: By replacing rigid grids with neural network function approximators, reachability analysis has jumped from 6D to 112D.
  • Stochastic Safety: For real-world systems where noise is inevitable, the Reachability Estimation Function (REF) provides a way to minimize the likelihood of failure, rather than just assuming a worst-case adversarial environment.
  • Shielding as a Safety Net: "Least restrictive" control laws (shielding) allow an RL agent to explore and learn freely, only intervening when the learned HJ value function indicates the agent is about to leave the "safe" reachable set.

Key Results

The paper summarizes a dramatic increase in the complexity of systems that can now be formally verified. As shown in the classification table, the field has moved from 2D-4D toy problems to 100+ dimensional lidar-based systems in less than a decade.

Classification of primary works in HJ reachability for RL

Notable qualitative successes include:

  • Sim-to-Real Transfer: Successfully deploying quadrupeds in real-world indoor spaces using HJ-based shielding trained in simulation.
  • Vision-Based Racing: Maintaining safety in high-speed autonomous racing using ego-vision inputs (40D representation).
  • Adversarial Robustness: Developing policies that remain safe even when facing worst-case disturbances or "antagonist" agents.

Limitations and Future Work

Despite the progress, several challenges remain:

  • Sample Efficiency: Learning a reliable safety certificate still requires many samples, which is problematic if the system must learn "on the fly" in the real world.
  • Catastrophic Forgetting: Neural networks can "forget" safe behaviors as they optimize for new rewards, potentially leading to sudden safety violations.
  • The Discounting Gap: There is still a theoretical discrepancy between the "true" safety set and the discounted versions used for stable learning.
  • Reentrance: Current HJ value functions are good at telling you if you are safe, but they don't always provide a clear path for how to return to the safe set once you've been pushed out.
  • Future Directions: The authors point toward lifelong learning (learning without resets) and the Koopman-Hopf framework (mapping nonlinear dynamics to linear spaces) as promising ways to further scale these methods.

Summary updated

Abstract

Recent literature has proposed approaches that learn control policies with high performance while maintaining safety guarantees. Synthesizing Hamilton-Jacobi (HJ) reachable sets has become an effective tool for verifying safety and supervising the training of reinforcement learning-based control policies for complex, high-dimensional systems. Previously, HJ reachability was restricted to verifying low-dimensional dynamical systems primarily because the computational complexity of the dynamic programming approach it relied on grows exponentially with the number of system states. In recent years, a litany of proposed methods addresses this limitation by computing the reachability value function simultaneously with learning control policies to scale HJ reachability analysis while still maintaining a reliable estimate of the true reachable set. These HJ reachability approximations are used to improve the safety, and even reward performance, of learned control policies and can solve challenging tasks such as those with dynamic obstacles and/or with lidar-based or vision-based observations. In this survey paper, we review the recent developments in the field of HJ reachability estimation in reinforcement learning that would provide a foundational basis for further research into reliability in high-dimensional systems.
BibTeX
@misc{ganai2024hamilton,
  author = {Milan Ganai and Sicun Gao and Sylvia L. Herbert},
  title = {{Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey}},
  year = {2024},
  eprint = {2407.09645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.09645},
}
Figure