Abstract
This paper proposed a novel PINN-based viscosity solution for HJB equations. Although there exists work using PINN to solve HJB, but none of them gives the solution in viscosity sense. This paper reveals the fact that using the convex neural network, one can guarantee the viscosity solution and thus the neural network can easily converge to the true solution of HJB despite of the starting point.
Introduction
In the field of optimal control, the Hamiltonian-Jacobi-Bellman (HJB) equation is considered the fundamental tool for determining optimal strategies. However, solving it numerically is notoriously difficult due to the "curse of dimensionality," where computational complexity grows exponentially with the system's variables.
Physics-Informed Neural Networks (PINNs) offer a promising mesh-free alternative, but they face a significant theoretical hurdle: HJB equations often have multiple "generalized" solutions. Only one of these—the viscosity solution—is the mathematically correct optimal cost function. Standard PINNs frequently converge to incorrect local minima or non-physical solutions unless they are initialized with a very accurate initial guess.
Methodology
The researchers propose a novel approach to force a PINN to find the viscosity solution by enforcing mathematical properties directly through the network's architecture and training curriculum.
Architectural Convexity
The core innovation is the use of an Input Convex Neural Network (ICNN). The authors demonstrate that for many control problems, the viscosity solution is inherently convex. Rather than relying on optimization penalties alone, they design a Partial Convex Neural Network that ensures the output is convex with respect to the system states ($x$) while remaining flexible with respect to time ($t$). This structural constraint effectively "locks out" the incorrect non-viscosity solutions that typically plague standard PINNs.
Strip Training Curriculum
To address the temporal dimension, the authors introduced a "strip training" method inspired by curriculum learning. Since the cost function's value is known at the final time ($T$), the network is first trained on a small "strip" of time near the end. Once mastered, the training moves backward toward time zero, using the previously learned regions to guide the optimization of the next strip. This prevents the network from drifting toward non-physical solutions during the training process.
Penalty-based Convexity (Alternative)
They also evaluated an alternative method involving a Hessian-based penalty term in the loss function to encourage convexity. However, the architectural approach (Method 1) proved significantly more robust in ensuring the network converged to the true viscosity solution.
Experimental Evaluation
The approach was evaluated on four synthetic nonlinear control benchmarks, including systems with quadratic costs and complex trigonometric functions designed to test the limits of the training method.
Key Results
- Global Convergence: The convex architecture successfully found the global optimum in highly nonlinear scenarios where penalty-based methods failed to converge.
- Robustness to Initialization: The combination of architectural convexity and strip training allowed the PINN to converge to the true solution even from random, distant starting points—a scenario where standard PINNs typically fail.
- High Accuracy: 3D error plots from the experiments demonstrate that the error remains consistently low (often $<0.01$) across the state space, even in complex nonlinear scenarios.
Conclusion
By hard-coding convexity into the neural network architecture, this research provides a reliable way to ensure PINNs converge to the mathematically correct viscosity solution of HJB equations. While challenges remain in scaling to extremely high-dimensional systems and handling problems where the solution is non-convex, this method represents a significant advancement in making deep learning-based optimal control more robust and theoretically sound.
Summary updated