In classical control theory, the "cost-to-go"—the total expected cost an agent will incur from its current state until the end of a task—is the North Star for decision-making. Mathematically, this cost is captured by the value function. If you know the value function, you can derive the optimal control law by simply moving in the direction that minimizes it.
However, finding this function requires solving the Hamilton-Jacobi-Bellman (HJB) equation, a partial differential equation (PDE) that suffers from the "curse of dimensionality." Traditional numerical methods rely on spatial grids; as the number of dimensions (degrees of freedom) increases, the number of grid points grows exponentially. A system with just 10 dimensions becomes computationally impossible for a standard computer to discretize.
A recent surge in research demonstrates that deep neural networks can bypass this curse by acting as grid-free value function approximators. By treating the value function as a continuous neural network rather than a discrete lookup table, researchers are now solving control problems in 100+ dimensions.
The Grid-Free Intuition
The core shift in this research direction is moving from discretization to approximation. Instead of trying to calculate the value at every point on a rigid grid, these methods use neural networks to learn the underlying structure of the value function from a finite set of samples.
As Xuewei Qi et al. demonstrate in their work on vehicle energy efficiency, neural networks allow control systems to handle continuous state spaces (like battery levels and power demand) with much finer granularity than traditional reinforcement learning. This scalability is even more pronounced in multi-agent systems. Derek Onken et al. show in A Neural Network Approach Applied to Multi-Agent Optimal Control that a neural network can parameterize the value function for a 50-agent swarm—a 150-dimensional problem—by training on a "cloud" of initial states rather than a fixed grid.
Solving PDEs as Optimization Problems
How do you train a neural network to satisfy a physics equation like the HJB? The most common approach is to turn the PDE into a loss function. This is often referred to as Physics-Informed Machine Learning.
In this framework, the neural network $V(x; \theta)$ takes a state $x$ as input and outputs the estimated cost-to-go. The training process involves minimizing the "residual" of the HJB equation—essentially, the network is penalized whenever its outputs violate the mathematical rules of optimal control. Manan Tayal et al. use this method in A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems to scale formal control methods to 20-dimensional environments, far beyond the 6-dimension limit of traditional solvers. Similarly, Yiming Meng et al. propose Physics-Informed Neural Network Policy Iteration, which solves nonlinear control problems by iteratively approximating a sequence of linear PDEs.
The Forward-Backward Stochastic Bridge
For systems involving noise or uncertainty, the HJB equation becomes even more complex. A powerful mechanism to solve these Stochastic Optimal Control (SOC) problems is to reformulate them as Forward-Backward Stochastic Differential Equations (FBSDEs).
In this view, the "forward" equation represents the system's trajectory through time, while the "backward" equation represents the evolution of the value function (or its gradient) along that trajectory. Marcus Pereira et al. show in Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs that neural networks can learn to solve these coupled equations simultaneously. This allows the agent to learn optimal policies in noisy environments while maintaining safety guarantees.
This FBSDE approach is highly versatile. It has been extended to:
- State Constraints: Bolun Dai et al. use LSTMs within the FBSDE framework to keep robots within physical boundaries.
- High-Dimensional Jumps: Patrick Cheridito et al. demonstrate in Deep learning for continuous-time stochastic control with jumps that these architectures can handle systems with sudden discontinuities in over 100 dimensions.
- Mean Field Control: Mo Zhou et al. leverage score dynamics to solve control problems for massive populations of interacting agents.
Bridging Reinforcement Learning and Control Theory
While Reinforcement Learning (RL) often struggles with the lack of mathematical guarantees, and Control Theory struggles with high dimensionality, neural value function approximation provides a bridge.
Insoon Yang et al. provide a rigorous foundation for this in Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time, treating the Q-function as a "viscosity solution" to an HJB equation. This allows for model-free algorithms that scale like deep learning but behave with the stability of classical control. By combining learned dynamics with HJB-based neural networks, researchers like Selim Engin et al. in Neural Optimal Control using Learned System Dynamics have achieved global control for complex systems like quadrotors without needing a pre-defined physics model.
Ultimately, by replacing rigid grids with flexible neural architectures, this field is transforming optimal control from a theoretical ideal into a practical tool for high-dimensional, real-time robotics.
Key Foundational Work
Deep reinforcement learning-based vehicle energy efficiency autonomous learning system (2017) — contributed 1 concept elements to this direction.
Learning to control partial differential equations: Regularized Fitted Q-Iteration approach (2016) — contributed 1 concept elements to this direction.