Deep learning method for solving stochastic optimal control problem via stochastic maximum principle

Shaolin Ji, Shige Peng, Ying Peng, Xichuan ZhangarXiv 2020 · OpenReview

TL;DR

This paper tackles the challenge of solving high-dimensional stochastic optimal control problems by leveraging deep learning and the stochastic maximum principle. They introduce a novel reformulation of the problem into an extended Hamiltonian system and propose three deep learning-based algorithms to solve it, demonstrating effectiveness particularly in high-dimensional scenarios.

What Problem Does This Paper Address?

Stochastic optimal control problems are fundamental in many fields, from finance to engineering. Traditionally, these problems are solved using Pontryagin's maximum principle or Bellman's dynamic programming. However, existing numerical methods struggle with high-dimensional problems due to the "curse of dimensionality," where computational complexity grows exponentially with the number of state variables. This paper aims to overcome this limitation by applying deep learning techniques, specifically by reformulating the problem through the lens of the stochastic maximum principle.

Methods

The core of the paper lies in reformulating the stochastic optimal control problem into a new control problem based on the stochastic maximum principle and its associated extended Hamiltonian system. This system is essentially a Forward-Backward Stochastic Differential Equation (FBSDE) with a maximum condition. Three deep learning-based algorithms are proposed to solve this reformulated problem:

  • Algorithm 1 (1-NNet): Uses a single neural network to simulate the control process. It treats time as an input and employs L-BFGS to approximate the optimal control in the maximum condition. This is noted as more suitable for low-dimensional cases.
  • Algorithm 2 (2-NNets): Addresses the computational burden of explicitly solving the maximum condition. It reformulates the condition as $H_u(t, x, u, p, q) = 0$ and uses two neural networks to simulate the control processes. This algorithm is more efficient for high-dimensional problems where the optimal control doesn't have an explicit solution.
  • Algorithm 3 (Explicit H): Assumes the Hamiltonian function $H$ is known and can be solved explicitly. This algorithm leverages deep learning for solving the resulting FBSDE and then uses the maximum condition to find the optimal control. This is presented as the most effective when an explicit form of $H$ is available.

The authors utilize deep neural networks (DNNs) where controls are treated as parameters to be learned via stochastic gradient descent.

Datasets

The paper does not introduce new datasets. Instead, it uses synthetic examples to demonstrate the performance of its proposed algorithms. These examples include:

  • A low-dimensional LQ (Linear Quadratic) control system.
  • High-dimensional LQ control problems.
  • A high-dimensional nonlinear control problem.
  • An application in calculating sub-linear expectation and nonlinear PDEs.
  • An example where the optimal control $\tilde{u}$ lacks an explicit solution.

Key Insights and Ideas

  • Reformulation via Stochastic Maximum Principle: The paper successfully reformulates the stochastic optimal control problem into a new variational problem solvable by deep learning, directly utilizing the stochastic maximum principle.
  • Deep Learning for High Dimensions: The proposed deep learning framework effectively tackles the "curse of dimensionality" that plagues traditional methods.
  • Three Algorithmic Approaches: The development of three distinct algorithms caters to different scenarios, offering flexibility based on whether the optimal control or the Hamiltonian has an explicit form.
  • Connection to Sub-linear Expectations and Nonlinear PDEs: The method's applicability extends beyond optimal control to calculating sub-linear expectations and solving fully nonlinear PDEs, highlighting its broader impact.
  • Criterion for Optimality: The reformulated problem provides a clear criterion (loss function approaching zero) to determine if a state-control pair is optimal.

Key Results

The numerical results across various examples demonstrate the effectiveness of the proposed algorithms:

  • Algorithm 3 is shown to be the most effective for the LQ control problem when an explicit form of $H$ is available, achieving very low relative errors (e.g., 0.003% for $n=5$).
  • For high-dimensional LQ problems ($n=100$), Algorithms 2 and 3 show comparable performance and significantly better stability and efficiency than Algorithm 1.
  • In cases where the optimal control $\tilde{u}$ lacks an explicit solution, Algorithm 2 proves to be a strong contender, outperforming Algorithm 1 in terms of computational cost and stability for high-dimensional problems.
  • The algorithms demonstrate good convergence for nonlinear control problems and sub-linear expectation calculations, with relative errors often in the order of $10^{-2}$ to $10^{-3}$ after sufficient iterations.
  • The paper also shows that their approach can achieve comparable results to existing deep learning methods for stochastic optimal control (e.g., Han and E [14]), while offering a more direct criterion for optimality.

Limitations and Future Work

  • Algorithm 1's Limitation: Algorithm 1 is noted as being difficult to apply to high-dimensional cases when the optimal control $\tilde{u}$ does not have an explicit solution due to the computational cost of solving the maximum condition in each iteration.
  • Non-convex Control Domains: While the paper focuses on convex control domains, Appendix A briefly discusses how the framework can be extended to non-convex domains by introducing second-order adjoint equations. This suggests future work in this direction.
  • Fully Coupled FBSDEs: The paper mentions that similar algorithms can be developed for systems described by fully coupled FBSDEs, indicating a potential area for further research.
  • Hyperparameter Tuning: As with most deep learning methods, the performance of the algorithms likely depends on careful tuning of hyperparameters (learning rate, network architecture, etc.), which is implicitly handled through experimentation.

Summary updated

Abstract

In this paper, we aim to solve the high dimensional stochastic optimal control problem from the view of the stochastic maximum principle via deep learning. By introducing the extended Hamiltonian system which is essentially an FBSDE with a maximum condition, we reformulate the original control problem as a new one. Three algorithms are proposed to solve the new control problem. Numerical results for different examples demonstrate the effectiveness of our proposed algorithms, especially in high dimensional cases. And an important application of this method is to calculate the sub-linear expectations, which correspond to a kind of fully nonlinear PDEs.
BibTeX
@misc{ji2020deep,
  author = {Shaolin Ji and Shige Peng and Ying Peng and Xichuan Zhang},
  title = {{Deep learning method for solving stochastic optimal control problem via stochastic maximum principle}},
  year = {2020},
  eprint = {2007.02227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.02227},
}
Figure