Neural Value Function Approximation for High Dimensional Control

31 concepts25 papers1 proposal

In classical control theory, the "cost-to-go"—the total expected cost an agent will incur from its current state until the end of a task—is the North Star for decision-making. Mathematically, this cost is captured by the value function. If you know the value function, you can derive the optimal control law by simply moving in the direction that minimizes it.

However, finding this function requires solving the Hamilton-Jacobi-Bellman (HJB) equation, a partial differential equation (PDE) that suffers from the "curse of dimensionality." Traditional numerical methods rely on spatial grids; as the number of dimensions (degrees of freedom) increases, the number of grid points grows exponentially. A system with just 10 dimensions becomes computationally impossible for a standard computer to discretize.

A recent surge in research demonstrates that deep neural networks can bypass this curse by acting as grid-free value function approximators. By treating the value function as a continuous neural network rather than a discrete lookup table, researchers are now solving control problems in 100+ dimensions.

The Grid-Free Intuition

The core shift in this research direction is moving from discretization to approximation. Instead of trying to calculate the value at every point on a rigid grid, these methods use neural networks to learn the underlying structure of the value function from a finite set of samples.

As Xuewei Qi et al. demonstrate in their work on vehicle energy efficiency, neural networks allow control systems to handle continuous state spaces (like battery levels and power demand) with much finer granularity than traditional reinforcement learning. This scalability is even more pronounced in multi-agent systems. Derek Onken et al. show in A Neural Network Approach Applied to Multi-Agent Optimal Control that a neural network can parameterize the value function for a 50-agent swarm—a 150-dimensional problem—by training on a "cloud" of initial states rather than a fixed grid.

Solving PDEs as Optimization Problems

How do you train a neural network to satisfy a physics equation like the HJB? The most common approach is to turn the PDE into a loss function. This is often referred to as Physics-Informed Machine Learning.

In this framework, the neural network $V(x; \theta)$ takes a state $x$ as input and outputs the estimated cost-to-go. The training process involves minimizing the "residual" of the HJB equation—essentially, the network is penalized whenever its outputs violate the mathematical rules of optimal control. Manan Tayal et al. use this method in A Physics-Informed Machine Learning Framework for Safe and Optimal Control of Autonomous Systems to scale formal control methods to 20-dimensional environments, far beyond the 6-dimension limit of traditional solvers. Similarly, Yiming Meng et al. propose Physics-Informed Neural Network Policy Iteration, which solves nonlinear control problems by iteratively approximating a sequence of linear PDEs.

The Forward-Backward Stochastic Bridge

For systems involving noise or uncertainty, the HJB equation becomes even more complex. A powerful mechanism to solve these Stochastic Optimal Control (SOC) problems is to reformulate them as Forward-Backward Stochastic Differential Equations (FBSDEs).

In this view, the "forward" equation represents the system's trajectory through time, while the "backward" equation represents the evolution of the value function (or its gradient) along that trajectory. Marcus Pereira et al. show in Safe Optimal Control Using Stochastic Barrier Functions and Deep Forward-Backward SDEs that neural networks can learn to solve these coupled equations simultaneously. This allows the agent to learn optimal policies in noisy environments while maintaining safety guarantees.

This FBSDE approach is highly versatile. It has been extended to:

Bridging Reinforcement Learning and Control Theory

While Reinforcement Learning (RL) often struggles with the lack of mathematical guarantees, and Control Theory struggles with high dimensionality, neural value function approximation provides a bridge.

Insoon Yang et al. provide a rigorous foundation for this in Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Time, treating the Q-function as a "viscosity solution" to an HJB equation. This allows for model-free algorithms that scale like deep learning but behave with the stability of classical control. By combining learned dynamics with HJB-based neural networks, researchers like Selim Engin et al. in Neural Optimal Control using Learned System Dynamics have achieved global control for complex systems like quadrotors without needing a pre-defined physics model.

Ultimately, by replacing rigid grids with flexible neural architectures, this field is transforming optimal control from a theoretical ideal into a practical tool for high-dimensional, real-time robotics.

Key Foundational Work

Deep reinforcement learning-based vehicle energy efficiency autonomous learning system (2017) — contributed 1 concept elements to this direction.
Learning to control partial differential equations: Regularized Fitted Q-Iteration approach (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_value_function_approximation_700eedeb,
  title = {neural value function approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-value-function-approximation/rdr_700eedeb019840aaae24b71e94c7fac2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_value_function_approximation_700eedeb_v1,
  title = {neural value function approximation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-value-function-approximation/rdr_700eedeb019840aaae24b71e94c7fac2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake