Recent Developments in Machine Learning Methods for Stochastic Control and Games

Ruimeng Hu, Mathieu Laurière

Abstract

Stochastic optimal control and games have found a wide range of applications, from finance and economics to social sciences, robotics, and energy management. Many real-world applications involve complex models which have driven the development of sophisticated numerical methods. Recently, computational methods based on machine learning have been developed for stochastic control problems and games. We review such methods, with a focus on deep learning algorithms that have unlocked the possibility to solve such problems even when the dimension is high or when the structure is very complex, beyond what is feasible with traditional numerical methods.

Introduction

At its core, stochastic control is about making optimal decisions in an uncertain, evolving environment. When multiple agents interact, it becomes a "game," where the goal is to find a Nash Equilibrium—a state where no one wants to change their strategy. The fundamental challenge in this field is the Curse of Dimensionality. Traditional numerical methods, such as finite difference schemes, require a grid that grows exponentially with the number of variables, making the management of high-dimensional systems (e.g., a portfolio of 100 stocks) computationally impossible. Furthermore, real-world systems often involve delays and common noise, adding layers of mathematical complexity that traditional tools cannot handle.

Methodological Framework

Recent developments have introduced several families of machine learning approaches to address these challenges:

  • Direct Parameterization: Strategies or controls are represented as deep neural networks. Stochastic Gradient Descent (SGD) is then used to minimize the total expected cost, either globally across the time horizon or locally via backward induction.
  • BSDE-based Methods (Deep BSDE): These reformulate the control problem as a Backward Stochastic Differential Equation. A neural network is trained to learn the relationship between the current state and the optimal future path by targeting correct terminal conditions.
  • PDE-based Methods (Deep Galerkin Method - DGM): These approximate solutions to the Hamilton-Jacobi-Bellman (HJB) equation—the "holy grail" of control theory—by minimizing the residual error of the PDE across the state space.
  • Deep Fictitious Play (DFP): This method decouples multi-agent games. Each player iteratively updates their strategy based on a "belief" about others' actions, transforming a complex multi-agent problem into a series of simpler individual control problems.

Applications and Empirical Models

The efficacy of these algorithms is often evaluated using synthetic benchmarks that represent real-world challenges. One prominent example is the use of epidemiological SEIR (Susceptible-Exposed-Infectious-Removed) models to determine optimal lockdown policies.

Epidemiological dynamics and lockdown policy optimization.

As shown in Figure 2, these methods can simultaneously model the lockdown policy $\ell(t)$ and the resulting dynamics of susceptible $S(t)$, exposed $E(t)$, and infectious $I(t)$ populations across different regions (NY, NJ, PA). Other applications include systemic risk models for inter-bank lending, optimal execution in finance, and malware propagation models in cybersecurity.

Numerical Results and Convergence

A key breakthrough of these methods is their ability to scale to 100+ dimensions, far exceeding the capabilities of traditional methods. In multi-agent scenarios, algorithms like Deep Fictitious Play have demonstrated remarkable accuracy, reaching Nash Equilibria with relative errors under 3% in just a few iterations.

Convergence of validation cost and stochastic path simulations.

Figure 5 illustrates the convergence behavior of these methods. The validation cost rapidly approaches the true cost as the number of Fictitious Play iterations increases, while the model successfully captures the underlying stochastic trajectories ($X_t$) and their conditional expectations ($m_t$).

Discussion and Future Directions

While deep learning has bridged the gap between high-dimensional mathematics and practical application, several hurdles remain. There are significant theoretical gaps regarding the convergence and generalization of these models in complex game scenarios. Additionally, performance is often sensitive to hyperparameter selection, and the models require high sample efficiency to be viable with noisy real-world data. Future work will likely focus on establishing unified benchmarks and developing more robust, sample-efficient architectures.

Summary updated

Abstract

Stochastic optimal control and games have found a wide range of applications, from finance and economics to social sciences, robotics and energy management. Many real-world applications involve complex models which have driven the development of sophisticated numerical methods. Recently, computational methods based on machine learning have been developed for stochastic control problems and games. We review such methods, with a focus on deep learning algorithms that have unlocked the possibility to solve such problems even when the dimension is high or when the structure is very complex, beyond what is feasible with traditional numerical methods. Here, we consider mostly the continuous time and continuous space setting. Many of the new approaches build on recent neural-network based methods for high-dimensional partial differential equations or backward stochastic differential equations, or on model-free reinforcement learning for Markov decision processes that have led to breakthrough results. In this paper we provide an introduction to these methods and summarize state-of-the-art works on machine learning for stochastic control and games.
BibTeX
@misc{hu2023recent,
  author = {Ruimeng Hu and Mathieu Laurière},
  title = {{Recent Developments in Machine Learning Methods for Stochastic Control and Games}},
  year = {2023},
  eprint = {2303.10257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.10257},
}
Figure