Neural Network Approaches for Parameterized Optimal Control

Deepanshu Verma, Nick Winovich, Lars Ruthotto, Bart van Bloemen Waanders

TL;DR

This paper compares two neural network strategies for solving optimal control problems that depend on uncertain parameters: a physics-heavy "Neural-HJB" approach and a data-driven Reinforcement Learning (RL) approach. The results show that by embedding the system's underlying physics into the learning process, the model-based approach achieves higher accuracy with 20x fewer simulations than state-of-the-art RL algorithms.

What Problem Does This Paper Address?

In many real-world scenarios—like robotics, autonomous driving, or managing environmental pollutants—we need to make optimal decisions in real-time. However, these systems often depend on parameters that are unknown or change over time (e.g., the wind speed affecting a drone or the exact source of a chemical leak).

Traditionally, solving these "optimal control" problems is computationally expensive. If the parameters change, you usually have to re-solve the entire problem from scratch. To fix this, researchers use amortization: they train a neural network offline to learn a "policy" that maps the current state and the current parameters to the best possible action.

The big challenge is the curse of dimensionality. When the state and parameters are high-dimensional (like a 2D grid of fluid temperatures), traditional math tools break down. This paper investigates whether it's better to use the "laws of physics" (model-based) or "trial and error" (data-driven RL) to train these high-dimensional controllers.

Methods

The authors compare two distinct training paradigms:

  1. Model-Based (Neural-HJB): This approach leverages the Hamilton-Jacobi-Bellman (HJB) equation—a partial differential equation (PDE) that defines the "value" of being in a certain state. The neural network is trained to minimize the "residual" of this equation. Crucially, it uses a feedback form: if you know the physics of the system, you can mathematically derive the optimal action directly from the gradient of the value function.
  2. Data-Driven (Reinforcement Learning): The authors use two "gold standard" RL algorithms: Proximal Policy Optimization (PPO) and Twin Delayed Deep Deterministic Policy Gradients (TD3). These are "actor-critic" methods where one network (the actor) tries out actions and another (the critic) evaluates them based on the rewards received, treating the underlying physics largely as a black box.

Datasets

The researchers evaluated these methods on a complex, synthetic benchmark: 2D Advection-Diffusion Control.

  • The Task: A contaminant is spreading in a 2D space. The controller must move a "sink" (a vacuum-like device) to prevent the contaminant from reaching a target zone.
  • The Parameters: The source location of the contaminant and the velocity of the "wind" (the advection field) are randomized.
  • Complexity: After discretization, the state and parameter space reached approximately 1,000 dimensions, far beyond what traditional non-neural methods can handle.

Key Insights and Ideas

  • Physics is a Shortcut: The Neural-HJB approach sees a "rapid drop" in loss at the very start of training. Because it uses the gradient information from the known dynamics, it doesn't have to "guess" as much as RL does.
  • RL's Exploration Problem: In high-dimensional spaces, RL spends a massive amount of time in "uninformed exploration." The RL agents often exhibited "unnecessary movement"—for example, drifting in the wrong direction before correcting—which increased their total cost.
  • Hyperparameter Sensitivity: RL was much harder to tune. The authors noted that RL required a "substantial number of system solves" just to find the right settings to prevent the training from becoming unstable or diverging.
  • Architecture Matters: Interestingly, the two methods preferred different architectures. RL performed best with Convolutional Neural Networks (CNNs) to process the spatial grid, while the HJB approach required simpler, smoother architectures (like ResNets with specific activations) to ensure the gradients needed for the feedback form were stable.

Key Results

  • Data Efficiency: The model-based HJB approach achieved accurate policies using 20x fewer PDE simulations than the RL approaches.
  • Accuracy: In the more difficult "sinusoidal" setup (complex wind patterns), the HJB approach reached a sub-optimality score of 0.03, while the RL models failed to get below 0.20 even after 15,000 simulations.
  • Reliability: During training, several RL models "diverged" (failed completely) after a few thousand solves, a phenomenon that never occurred with the model-based HJB approach.

Limitations and Future Work

  • Requirement of a Model: The HJB approach's biggest strength is also its limitation: it requires a well-defined, differentiable mathematical model of the system. If the physics are unknown (a "true" black box), RL remains the only viable option.
  • Feedback Form Complexity: The HJB method relies on a closed-form expression for the optimal control. While common in many problems, some complex constraints might make this expression difficult to derive.
  • Future Directions: The authors plan to test these methods on even more complex, non-linear dynamics and investigate how these neural controllers scale as the grid resolution (discretization) becomes even finer.

Summary updated

Abstract

We consider numerical approaches for deterministic, finite-dimensional optimal control problems whose dynamics depend on unknown or uncertain parameters. We seek to amortize the solution over a set of relevant parameters in an offline stage to enable rapid decision-making and be able to react to changes in the parameter in the online stage. To tackle the curse of dimensionality arising when the state and/or parameter are high-dimensional, we represent the policy using neural networks. We compare two training paradigms: First, our model-based approach leverages the dynamics and definition of the objective function to learn the value function of the parameterized optimal control problem and obtain the policy using a feedback form. Second, we use actor-critic reinforcement learning to approximate the policy in a data-driven way. Using an example involving a two-dimensional convection-diffusion equation, which features high-dimensional state and parameter spaces, we investigate the accuracy and efficiency of both training paradigms. While both paradigms lead to a reasonable approximation of the policy, the model-based approach is more accurate and considerably reduces the number of PDE solves.
BibTeX
@inproceedings{verma2024neural,
  author = {Deepanshu Verma and Nick Winovich and Lars Ruthotto and Bart van Bloemen Waanders},
  title = {{Neural Network Approaches for Parameterized Optimal Control}},
  booktitle = {Foundations},
  year = {2024},
  doi = {10.3934/fods.2024042},
}
Figure