Learning Value Functions by Minimizing Hamilton Jacobi Bellman Residuals

26 concepts19 papers1 proposal

To find the optimal action for a robot, a chemical plant, or an autonomous vehicle, you need to know the "value" of your current state—the total future cost or reward you will accumulate from this moment onward. In continuous-time optimal control, this value function is governed by a fundamental partial differential equation (PDE) known as the Hamilton-Jacobi-Bellman (HJB) equation.

For decades, solving the HJB equation meant relying on grid-based numerical methods. But grids suffer from the curse of dimensionality: if you double the number of state variables, the number of grid points scales exponentially. Controlling a system with 10, 50, or 100 dimensions becomes computationally impossible.

A rapidly growing body of research bypasses this limitation by combining the structural guarantees of classical control theory with the flexibility of deep learning. Instead of using a grid, these methods parameterize the value function as a neural network and train it by directly minimizing how much it violates the HJB equation. This approach turns a notoriously difficult PDE-solving problem into a scalable, self-supervised optimization task.

The Intuition: The HJB Equation as a Loss Function

To understand why this works, consider what the HJB equation actually asserts. At its core, it is a continuous-time statement of optimality: the time derivative of the true value function, plus the minimized Hamiltonian (which represents the immediate cost combined with the system's underlying dynamics), must equal exactly zero.

$$\partial_t V(s, t) + \min_u \mathcal{H}\left(s, u, \nabla_s V(s, t)\right) = 0$$

If we approximate the true value function using a neural network $V_\theta(s, t)$, we can plug it directly into this equation. Because the network's weights $\theta$ are initially random, the left side of the equation will not equal zero. It will yield a leftover error, known as the HJB residual.

By leveraging automatic differentiation, we can compute the exact gradients of the neural network with respect to its inputs ($\partial_t V_\theta$ and $\nabla_s V_\theta$) at any arbitrary state. We can then define a loss function that penalizes the square of this residual. Minimizing this loss forces the neural network to conform to the physical laws and optimality principles governing the system, acting as a powerful self-supervised constraint.

Defeating the Curse of Dimensionality

Because neural networks are mesh-free function approximators, they do not require a rigid grid. Instead, researchers use collocation methods: they sample states randomly across the state space or along trajectories, evaluate the HJB residual at those points, and update the network weights.

This shift allows formal control and verification to scale to high-dimensional systems that were previously intractable. For example, Somil Bansal et al. demonstrated in DeepReach: A Deep Learning Approach to High-Dimensional Reachability that minimizing Hamilton-Jacobi-Isaacs residuals allows neural networks to compute safe controllers and verify systems with more than 10 dimensions, such as multi-vehicle avoidance scenarios.

When scaled even further, this framework can coordinate massive swarms. Derek Onken et al. leveraged HJB penalties alongside Pontryagin's Maximum Principle in A Neural Network Approach Applied to Multi-Agent Optimal Control and A Neural Network Approach for High-Dimensional Optimal Control Applied to Multiagent Path Finding. By penalizing deviations from the HJB optimality condition during training, the authors successfully navigated a 50-agent swarm—a 150-dimensional state space—through complex obstacles in real-time.

Grounding Reinforcement Learning in Physical Laws

In standard model-free reinforcement learning (RL), a "critic" network learns the value function by observing discrete reward signals over time. However, this trial-and-error approach often requires millions of samples and struggles to respect continuous physical boundaries.

By treating the critic as a Physics-Informed Neural Network (PINN) that minimizes the continuous-time HJB residual, the learning process becomes deeply anchored in the system's known dynamics.

Standard RL Critic:      [Sampled Transitions] ----> Temporal Difference Error ----> Update

Physics-Informed Critic:  [Continuous Dynamics] ----> Analytical HJB Residual  ----> Update

This integration bridges the gap between data-driven control and physical reality:

Regularization and Policy Iteration

Even when external data or expert demonstrations are available, HJB residual minimization serves as an excellent regularizer. Instead of simply interpolating between training data points, the network is forced to learn the underlying physics of the control landscape.

For instance, Siyuan Xu et al. demonstrated in Meta Value Learning for Fast Policy-Centric Optimal Motion Planning that incorporating the HJB equation as a constraint within a meta-learning loss function prevents overfitting and enables robots to adapt to new motion planning tasks in seconds. Similarly, Yiming Meng et al. embedded the Generalized HJB equation directly into a policy iteration framework in Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification, allowing a model to approximate the value function across the entire state space without requiring pre-computed labels from an external solver.

Beyond Deterministic Systems

The utility of HJB residual minimization extends far beyond deterministic, fully observable systems. By modifying the underlying PDE, researchers have adapted this framework to handle uncertainty, partial observability, and population dynamics:

  • Stochastic Systems and Mean Field Control: When systems are subject to random noise, the HJB equation gains a second-order derivative term, becoming a "viscous" HJB equation. In Score-based Neural Ordinary Differential Equations for Computing Mean Field Control Problems, Mo Zhou et al. integrated viscous HJB residuals into the loss function of score-based neural ODEs. This physics-based inductive bias yielded an order-of-magnitude improvement in tracking how large populations evolve over time.
  • Partially Observable Environments: In POMDPs in Continuous Time and Discrete Spaces, Bastian Alt et al. derived an HJB partial integro-differential equation that operates over a continuous-time belief space. They trained neural networks offline by minimizing this residual error, providing a robust mechanism for optimal decision-making when observations arrive at arbitrary times.
  • Generative Modeling: Even in population biology and trajectory reconstruction, the HJB equation provides a foundational constraint. Takeshi Koshizuka et al. utilized an HJB-derived loss in Neural Lagrangian Schrödinger Bridge: Diffusion Modeling for Population Dynamics to ensure that the learned potential landscapes guiding cell or organism trajectories remain mathematically consistent with the defined cost landscape.

By transforming the fundamental equations of optimal control into self-supervised loss functions, HJB residual minimization allows neural networks to act not just as black-box statistical interpolators, but as mathematically rigorous, high-dimensional solvers for complex physical systems.

Key Foundational Work

POMDPs in Continuous Time and Discrete Spaces (2020) — contributed 2 concept elements to this direction.
DeepReach: A Deep Learning Approach to High-Dimensional Reachability (2020) — contributed 1 concept elements to this direction.
Learning Deep Stochastic Optimal Control Policies Using Forward-Backward SDEs (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hjb_residual_minimization_30b2accf,
  title = {HJB residual minimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hjb-residual-minimization/rdr_30b2accf0c35495dbb88a2c59a12944b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hjb_residual_minimization_30b2accf_v1,
  title = {HJB residual minimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hjb-residual-minimization/rdr_30b2accf0c35495dbb88a2c59a12944b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake