Authors: Nikolas Nüsken and Lorenz Richter (2021)
Topic: High-dimensional PDE solvers, optimal control, and machine learning.
Introduction
Hamilton-Jacobi-Bellman (HJB) equations are a fundamental class of partial differential equations (PDEs) that arise in optimal control, reinforcement learning, and rare-event simulation. Despite their importance, solving them in high-dimensional settings remains a significant challenge. Traditional grid-based methods suffer from the "curse of dimensionality," where computational costs scale exponentially with the number of variables. Furthermore, systems exhibiting metastability—such as molecular dynamics where transitions between states are rare—pose additional difficulties for standard sampling techniques.
In this work, Nüsken and Richter propose a novel framework that reformulates the HJB problem through the lens of controlled diffusions and probability measures on path space. By leveraging deep neural networks and a new divergence metric, they provide a robust method for solving these PDEs in high dimensions.
Theoretical Framework: Path Space and Divergences
The authors move away from treating the HJB equation as a purely analytical PDE. Instead, they frame the problem as finding an optimal control strategy that minimizes a specific cost functional. This is equivalent to minimizing the "distance" between two probability measures on the space of all possible system trajectories (path space):
- The Reference Measure ($\mathbb{P}$): The distribution of trajectories under the uncontrolled, stochastic dynamics.
- The Target Measure ($\mathbb{Q}$): The distribution of trajectories that optimally satisfy the control objective.
The Log-Variance Divergence
The core innovation of the paper is the introduction of the log-variance divergence loss function:
$$D_{\mathbb{P}}^{\text{Var(log)}}(\mathbb{P}_1 | \mathbb{P}_2) = \text{Var}_{\mathbb{P}}\left(\log \frac{d\mathbb{P}_2}{d\mathbb{P}_1}\right)$$
Unlike standard Kullback-Leibler (KL) divergence or cross-entropy losses, which are commonly used in variational inference and reinforcement learning, the log-variance loss offers several distinct advantages for HJB equations:
- Robustness under Tensorization: The authors prove that while the relative error of gradients in standard losses often grows exponentially with the dimension $d$, the log-variance loss remains controlled. This property is essential for scaling to high-dimensional systems.
- Zero Variance at the Optimum: As the neural network approaches the true optimal control, the variance of the estimator goes to zero. This leads to exceptional stability in the final stages of training, where other Monte Carlo-based methods often suffer from persistent noise.
- Shift Invariance: The loss is naturally invariant to additive constants in the value function. This allows the neural network to focus on learning the gradient of the solution (the control strategy) rather than the absolute value of the cost.
Methodology: Iterative Diffusion Optimization (IDO)
The authors implement their framework using Iterative Diffusion Optimization (IDO). The control strategy $u$ is parameterized by a deep neural network (specifically a DenseNet architecture). The training process involves simulating trajectories of the stochastic differential equation (SDE) and updating the network weights to minimize the log-variance divergence.
By using neural networks, the model can approximate complex, high-dimensional functions that would be impossible to represent on a grid. The iterative nature of the algorithm allows the sampling distribution to gradually shift toward the optimal trajectories, effectively focusing computational effort on the most relevant regions of the state space.
Experimental Results
The framework was evaluated on several challenging synthetic systems designed to test scalability and robustness:
1. High-Dimensional Scaling (OU and LQG Dynamics)
In tests involving Ornstein-Uhlenbeck (OU) dynamics and Linear Quadratic Gaussian (LQG) control in up to 40 dimensions, the log-variance loss demonstrated rapid convergence. While standard "moment losses" used in previous Deep BSDE (Backward Stochastic Differential Equation) solvers struggled with slow convergence and high gradient variance, the proposed method remained stable.
2. Metastable Systems (Double-Well Potential)
The most significant result occurred in the 10-dimensional metastable double-well experiment. In this scenario, a particle must cross a high-energy barrier— a rare event that standard sampling methods often miss. The log-variance loss was the only method capable of accurately approximating the optimal control. Other divergence metrics were "frustrated" by the high variance of gradients near the transition barriers, failing to learn the necessary steering force to overcome the potential energy gap.
Conclusion and Future Directions
Nüsken and Richter provide a rigorous theoretical foundation for using log-variance divergences to solve HJB PDEs. Their approach successfully addresses the curse of dimensionality and the challenges of metastability, outperforming existing deep learning-based PDE solvers in stability and accuracy.
Future work may involve extending this framework to more general nonlinear PDEs where shift invariance does not naturally hold, or applying these path-space techniques to Bayesian variational inference. The success of the log-variance loss suggests that focusing on variance reduction in gradient estimation is a powerful strategy for high-dimensional stochastic optimization.
Summary updated