TL;DR
This paper tackles the challenge of solving high-dimensional stochastic optimal control problems by leveraging the stochastic maximum principle and deep learning. They reformulate the problem into a new, more tractable form and propose three novel deep learning algorithms to solve it, demonstrating effectiveness even when optimal controls are not explicitly known.
What Problem Does This Paper Address?
Stochastic optimal control problems are fundamental in many fields, from finance to engineering. Traditionally, these problems are solved using Pontryagin's maximum principle or Bellman's dynamic programming. However, these methods, and many existing numerical approaches, struggle with "the curse of dimensionality" – their computational complexity explodes as the number of state variables increases. This paper aims to overcome this limitation by applying deep learning techniques to solve these problems, specifically by reformulating them through the lens of the stochastic maximum principle.
Methods
The core of the paper's approach is to reformulate the stochastic optimal control problem using the stochastic maximum principle (SMP). This involves introducing an "extended Hamiltonian system," which is essentially a Forward-Backward Stochastic Differential Equation (FBSDE) with an added maximum condition. This reformulation leads to a new control problem with a simpler quadratic terminal cost, making it easier to verify optimality.
The authors then propose three distinct deep learning algorithms to solve this reformulated problem:
- Algorithm 1 (1-NNet): This algorithm uses a single neural network to simulate the control process. It's suitable when the optimal control has an explicit solution, but can become computationally expensive for high-dimensional problems if the optimal control is not explicit.
- Algorithm 2 (2-NNets): This algorithm is designed for cases where the optimal control doesn't have an explicit solution. It uses two neural networks and reformulates the maximum condition into a constraint, making it more efficient for high-dimensional problems.
- Algorithm 3 (Explicit H): This algorithm is for situations where the Hamiltonian function (which defines the optimal control) has a known, explicit expression. It leverages deep learning to solve the underlying FBSDE, and then the optimal control is found directly from the explicit Hamiltonian.
All algorithms utilize deep neural networks to approximate the control processes and are trained using stochastic gradient descent methods.
Datasets
The paper does not explicitly mention or introduce new datasets for training. Instead, it uses simulated data generated from the specific stochastic optimal control problems it sets up for numerical experiments. These problems serve as benchmarks for evaluating the proposed algorithms.
Key Insights and Ideas
- Reformulation is Key: The paper demonstrates that reformulating the stochastic optimal control problem via the stochastic maximum principle into an FBSDE with a maximum condition is a powerful strategy for enabling deep learning solutions.
- Deep Learning for High Dimensions: The proposed deep learning framework effectively tackles the "curse of dimensionality" that plagues traditional methods.
- Flexibility for Unknown Controls: Algorithm 2 is particularly noteworthy for its ability to handle problems where the optimal control cannot be analytically derived, a common scenario in complex systems.
- Sub-linear Expectations and PDEs: The method has a direct application in computing sub-linear expectations, which are closely related to fully nonlinear partial differential equations, opening doors to solving a broader class of problems.
Key Results
The paper presents numerical results for various examples, including low-dimensional and high-dimensional cases.
- Algorithm 3 is shown to be the most effective for problems where the optimal control has an explicit solution, achieving very low relative errors (e.g., 0.003% for n=5) with significantly reduced computation time compared to other algorithms.
- For high-dimensional problems where the optimal control is not explicit, Algorithms 2 and 3 demonstrate much more effective performance than Algorithm 1, with relative errors generally below 1%.
- The algorithms are shown to converge well, with cost functional values closely matching benchmarks (e.g., ODE45 solutions) in both low and high-dimensional LQ control problems.
- In a nonlinear control problem, Algorithm 2 and 3 achieved close results to a Monte Carlo simulation benchmark.
- For a problem involving sub-linear expectations, Algorithm 3 achieved a relative error of approximately 0.02% after 5000 iterations.
- When comparing with a prior deep learning method (Han and E [14]), the proposed Algorithm 2 showed comparable cost functional values and lower variances.
Limitations and Future Work
- Algorithm 1's Limitation: Algorithm 1 is noted as being difficult to apply to high-dimensional cases when the optimal control is not explicit due to the computational cost of solving the maximum condition in each iteration.
- Non-convex Control Domains: The paper primarily focuses on convex control domains. While Appendix A briefly mentions extending the framework to non-convex domains, this is an area for further exploration.
- Fully Coupled FBSDEs: The paper suggests that the framework can be extended to handle fully coupled FBSDEs, which is a direction for future research.
- Algorithm Choice: The paper provides guidance on choosing the appropriate algorithm based on whether the optimal control is explicit and the nature of the Hamiltonian function. Further research could explore more adaptive or unified algorithmic approaches.
Summary updated