Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control

Jeongho Kim, Insoon YangarXiv 2020 · arXiv

Abstract

This paper introduces a theoretical framework for maximum entropy optimal control in continuous-time systems. By deriving a novel "Soft" Hamilton-Jacobi-Bellman (HJB) equation, the authors bridge the gap between discrete-time maximum entropy reinforcement learning (RL) and continuous-time control theory. The framework utilizes a relaxed control formulation to encourage exploration, resulting in smoother value functions and more robust controllers. The authors provide tractable solvers for both linear-quadratic and nonlinear systems, demonstrating significant improvements in sample efficiency and learning speed compared to standard adaptive dynamic programming methods.

Introduction

In reinforcement learning (RL), maximum entropy methods—most notably Soft Actor-Critic—have established themselves as a gold standard for discrete-time systems. By augmenting the reward objective with an entropy term, these methods promote robust exploration and prevent premature convergence to suboptimal policies. However, extending these benefits to continuous-time systems, such as robotics, drones, and industrial plants, has historically been challenging.

Standard optimal control for continuous-time deterministic systems typically seeks a single optimal trajectory. This often results in non-smooth value functions that are computationally difficult to solve and lack the inherent exploration properties required for efficient learning in unknown environments. This research addresses this gap by providing a rigorous theoretical foundation that combines the probabilistic nature of maximum entropy with the differential equations governing continuous-time physics.

Theoretical Framework: The Soft HJB Equation

The core innovation of this work is the derivation of the Soft Hamilton-Jacobi-Bellman (HJB) equation. The authors achieve this through several key steps:

Relaxed Control Formulation

Instead of the traditional approach of selecting a single control input $u(t)$, the authors employ a "relaxed" control formulation. In this setting, the controller selects a probability distribution over actions. This shift allows the system to incorporate entropy regularization directly into the objective function.

The Soft Maximum Operator

In the standard HJB equation, a "min" or "max" operator is used to determine the optimal control. The authors replace this with a "soft maximum" operator—specifically, an integral of an exponential. They rigorously prove that the resulting optimal value function is the unique viscosity solution to this Soft HJB equation.

Statistical Mechanics Connection

The paper reveals a profound link between optimal control and statistical mechanics. The "Soft" Hamiltonian corresponds to the negative Helmholtz free energy, and the resulting optimal control policy naturally takes the form of a Boltzmann distribution. This connection provides a physical intuition for the exploration-exploitation trade-off governed by the "temperature" parameter $\alpha$.

Computational Methods and Solvers

To make the theory applicable to real-world problems, the authors developed several tractable numerical solvers:

  1. Linear-Quadratic (LQ) Systems: For LQ problems, the Soft HJB equation reduces to a modified Riccati equation, allowing for efficient computation similar to standard LQR methods.
  2. Nonlinear Systems: For more complex dynamics, the authors utilize generalized Hopf–Lax formulas. These formulas enable "grid-free" numerical solutions, effectively bypassing the "curse of dimensionality" that typically hinders grid-based methods in high-dimensional state spaces.
  3. Adaptive Dynamic Programming (ADP): The authors developed both on-policy and off-policy model-free RL algorithms. These allow the system to learn optimal "soft" controllers even when the underlying system dynamics are entirely unknown.

Experimental Evaluation

The framework was evaluated using simulated dynamical systems to test both the nonlinear solvers and the learning algorithms.

  • Van der Pol Oscillator: This classic nonlinear system served as a benchmark for the grid-free Hopf-Lax solver, demonstrating its ability to handle complex dynamics without a state-space grid.
  • High-Dimensional Linear Systems: The authors tested the ADP algorithms on 10-dimensional and 20-dimensional systems to compare performance against standard baselines.

Key Results

The experimental results highlight the efficiency of the maximum entropy approach:

  • Sample Efficiency: In a 10D linear system, the proposed method required only 155 data points to satisfy the rank conditions for learning, whereas the standard adaptive DP method required 431.5 points.
  • Learning Speed: The maximum entropy method completed the learning phase in 9.30 seconds, nearly twice as fast as the baseline's 17.26 seconds.
  • Control Performance: The "soft" controllers achieved lower total running costs and faster settling times (6.60s vs 8.36s), suggesting that principled exploration leads to the discovery of superior control policies.

Discussion and Insights

Beyond the quantitative gains, the paper offers several theoretical insights:

  • Improved Regularity: Entropy regularization acts as a smoothing mechanism. The resulting value functions are more differentiable than those in standard optimal control, simplifying the synthesis of controllers.
  • Gaussian Emergence: For control-affine systems with quadratic costs, the paper proves that the optimal policy is naturally a Gaussian distribution. This provides a formal theoretical justification for the common RL practice of using Gaussian noise for exploration.
  • Asymptotic Consistency: The authors demonstrate that as the temperature parameter $\alpha$ approaches zero, the Soft HJB equation recovers the standard HJB equation, ensuring the framework is a true generalization of classical theory.

Limitations and Future Work

While the paper provides a robust foundation for deterministic systems, several areas remain for future exploration. The current work focuses on deterministic physics; extending the Soft HJB framework to stochastic differential equations (SDEs) is a necessary next step for systems with inherent noise. Additionally, the Hopf-Lax formulas rely on the existence of bi-characteristic curves, which may only exist locally in highly complex Hamiltonians, potentially limiting the time horizon for certain nonlinear solvers. Finally, the temperature parameter $\alpha$ currently requires manual tuning, and automating this process for unknown systems remains an open challenge.

Summary updated

Abstract

Maximum entropy reinforcement learning (RL) methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of existing techniques are designed for discrete-time systems. As a first step toward their extension to continuous-time systems, this paper considers continuous-time deterministic optimal control problems with entropy regularization. Applying the dynamic programming principle, we derive a novel class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. Our maximum entropy formulation is shown to enhance the regularity of the viscosity solution and to be asymptotically consistent as the effect of entropy regularization diminishes. A salient feature of the HJB equations is computational tractability. Generalized Hopf-Lax formulas can be used to solve the HJB equations in a tractable grid-free manner without the need for numerically optimizing the Hamiltonian. We further show that the optimal control is uniquely characterized as Gaussian in the case of control affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. Lastly, we discuss how to extend our results to continuous-time model-free RL by taking an adaptive dynamic programming approach. To our knowledge, the resulting algorithms are the first data-driven control methods that use an information theoretic exploration mechanism in continuous time.
BibTeX
@misc{kim2020hamilton,
  author = {Jeongho Kim and Insoon Yang},
  title = {{Hamilton-Jacobi-Bellman Equations for Maximum Entropy Optimal Control}},
  year = {2020},
  eprint = {2009.13097},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.13097},
}
Figure