The Bottleneck: The Hamilton-Jacobi-Bellman Equation and the Grid
To understand why high-dimensional optimal control is notoriously difficult, we must look at how classical methods solve for the optimal policy. In continuous-time stochastic optimal control, the standard mathematical framework for finding a globally optimal policy relies on the Hamilton-Jacobi-Bellman (HJB) equation. The HJB equation is a partial differential equation (PDE) that determines a value function $V(x)$ across the entire state space.
If a system has a state vector $x \in \mathbb{R}^d$, traditional numerical methods (such as finite-difference or grid-based discretization schemes) divide each dimension into $M$ discrete intervals. This requires solving the PDE across $M^d$ grid points. As Insoon Yang et al. note in Path integral formulation of stochastic optimal control with generalized costs, this exponential scaling makes classical grid-based solvers completely intractable for complex, high-dimensional engineering systems. For instance, a robot with dozens of degrees of freedom or a large-scale financial portfolio cannot be discretized this way. Similarly, Zhenghong Yu et al. show in Worst-case formulations of model predictive control for systems with bounded parameters that closed-loop dynamic programming formulations quickly become computationally prohibitive as the state dimension grows.
To bypass this bottleneck, modern research focuses on grid-free methods. Instead of solving a PDE globally over an entire space, these methods evaluate the system along trajectories, using deep learning, path integrals, or sampling to generalize across high-dimensional spaces.
Trajectory-Based Reformulations: Path Integrals and FBSDEs
One powerful intuition for avoiding the grid is to transform the global PDE into an expectation over local stochastic trajectories. Under certain conditions—specifically when the control cost is quadratic and the noise enters the system linearly—the nonlinear HJB equation can be linearized via a logarithmic transformation (the Cole-Hopf transformation). This leads to path integral control, where the optimal control policy can be computed by taking a weighted average over simulated trajectories. Insoon Yang et al. extended these path integral formulations to generalized costs by augmenting the state space, making it applicable to complex engineering systems without relying on rigid spatial grids.
Another trajectory-based approach relies on the Feynman-Kac formula, which links second-order PDEs (like the HJB equation) to systems of Forward-Backward Stochastic Differential Equations (FBSDEs). In this formulation:
- The forward SDE describes the physical dynamics of the system under noise.
- The backward SDE propagates the value function and its gradient (which corresponds to the optimal control law) backward in time from the terminal reward.
By simulating trajectories forward and backward, researchers can optimize control policies along the paths the system actually visits, rather than over an arbitrary grid. Marcus A. Pereira et al. leverage this in Deep Forward-Backward SDEs for Min-max Control, using recurrent neural networks to stabilize min-max control problems in high dimensions. Similarly, Ziyi Wang et al. demonstrate in Learning Deep Stochastic Optimal Control Policies Using Forward-Backward SDEs that this connection allows deep networks to learn stochastic policies for high-dimensional robotic systems without grid discretization.
Deep Learning and Operator Learning as Global Approximators
While trajectory-based methods solve the control problem locally along sampled paths, deep neural networks can act as high-dimensional function approximators to reconstruct the global value function or policy. Because neural networks scale well to high-dimensional inputs, they can break the curse of dimensionality.
For example, Yiming Meng et al. introduce a physics-informed neural network (PINN) approach in Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification, embedding the HJB physics directly into the loss function to guarantee convergence and verification on complex platforms. To combine global optimality with local efficiency, Derek Onken et al. in A Neural Network Approach for High-Dimensional Optimal Control Applied to Multiagent Path Finding fuse the global benefits of the HJB equation with the local efficiency of Pontryagin's Maximum Principle (PMP), enabling real-time feedback control for systems with over a hundred dimensions. In Learning Stochastic Optimal Policies via Gradient Descent, Michael Poli derives a path-wise backward adjoint sensitivity method for SDEs, allowing efficient gradient computation directly in high-dimensional parameter spaces.
Going a step further, Jae Yong Lee proposes using deep operator learning in Hamilton-Jacobi Based Policy-Iteration via Deep Operator Learning. Instead of learning a value function for a single specific control problem, operator learning maps entire functional spaces (such as varying cost functions or dynamics) to their corresponding optimal control policies, bypassing traditional grid-based limitations entirely.
Generative Modeling and Matching Frameworks
A recent paradigm shifts the problem from solving PDEs to matching probability distributions, drawing inspiration from generative AI models like diffusion and flow matching.
In stochastic control, we often want to steer an entire distribution of states from an initial profile to a target profile optimally. Carles Domingo-Enrich et al. introduce Stochastic Optimal Control Matching (SOCM) in Stochastic Optimal Control Matching. By adapting the least-squares training objectives of diffusion models and using a path-wise reparameterization trick, the authors dramatically reduce gradient variance, allowing standard neural networks to solve high-dimensional stochastic control problems with high accuracy.
Similarly, Yuhang Mei et al. develop Flow matching for stochastic linear control systems, which derives feedback control laws that steer probability distributions through specific control channels. This bridges generative modeling and classical control, offering scalable neural network approximations for high-dimensional state spaces where solving the underlying partial differential equations would otherwise be impossible.