Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime

Bekzhan Kerimkulov, James-Michael Leahy, David Siska, Lukasz Szpruch

Introduction

In modern Reinforcement Learning (RL), policy gradient methods are a cornerstone of empirical success, particularly when paired with deep neural networks to handle continuous state and action spaces. However, a significant gap has long existed between this empirical success and our theoretical understanding. While convergence is well-understood in "tabular" settings—where states and actions are finite—the continuous case introduces a non-convex optimization landscape that is notoriously difficult to analyze.

Presented at ICML 2022, this research provides a rigorous mathematical foundation for policy gradient methods in continuous environments. By shifting the analytical perspective from individual neural network weights to a "mean-field" distribution of weights, the authors prove that specific regularization techniques can guarantee exponential convergence to a global optimum.

The Challenge: Non-Convexity in Continuous Spaces

The primary obstacle in proving the convergence of neural-network-based RL is the non-convexity of the objective function. In standard optimization, the "landscape" of a neural network is often filled with local optima and saddle points that can trap gradient-based algorithms. Previous research into the "mean-field" regime—where the neural network is considered in the limit of infinite width—suggested that if an algorithm converges, it reaches a global optimum. However, these studies stopped short of proving that convergence actually occurs or determining the speed of that convergence. This paper addresses these two critical unknowns.

Methodology: The Mean-Field Framework

To tackle the problem of non-convexity, the authors employ a suite of sophisticated mathematical tools that transform the optimization problem into a more manageable form.

1. Lifting to the Mean-Field Regime

Instead of tracking the evolution of a finite set of weights $\theta$, the authors analyze the distribution of weights $\nu$. In the limit of an infinitely wide, one-hidden-layer neural network, the optimization problem "lifts" into the space of probability measures. In this higher-dimensional space, the objective function becomes much better behaved.

2. Double Regularization

The core innovation of the paper is the application of "Double Regularization":

  • Policy Entropy ($\tau$): This is a standard technique in RL (used in algorithms like Soft Actor-Critic) that encourages the agent to explore by penalizing overly deterministic policies.
  • Parameter Entropy ($\sigma$): This is a novel addition in this context. It regularizes the distribution of the neural network weights themselves relative to a prior distribution. This second layer of regularization is the "secret sauce" that smooths the optimization landscape.

3. Wasserstein Gradient Flow and PDE Analysis

The learning process is modeled as a Wasserstein Gradient Flow, describing how the weight distribution $\nu$ moves through the landscape toward the optimum. By using the nonlinear Fokker–Planck–Kolmogorov (FPK) equation, the authors describe the dynamics of this distribution over time, allowing them to prove both stability and specific convergence rates.

Key Insights and Results

Lifting to Convexity

The research demonstrates that while the objective is non-convex with respect to individual weights, it can be made strictly concave when viewed as a function of the weight distribution $\nu$. This transformation is essential for guaranteeing that the gradient flow always moves toward a global maximum.

Global Convergence and Exponential Rates

The paper’s primary contribution is Theorem 2.12, which establishes that under sufficient parameter regularization (where $\sigma$ is large enough), the weight distribution $\nu_t$ is guaranteed to converge to a unique global maximizer $\nu^*$. Furthermore, this convergence is fast; the distance to the optimum shrinks at an exponential rate $O(e^{-\beta t})$.

The "Posterior" Optimum

The authors reveal an elegant structural result: the optimal weight distribution $\nu^*$ takes the form of a Gibbs distribution. This mirrors the structure of a posterior in Bayesian inference, suggesting a deep link between optimal policy approximation and Bayesian principles.

Limitations and Future Directions

While this work provides a landmark theoretical result, it includes several caveats that define the roadmap for future research:

  • The Large $\sigma$ Requirement: The proof for exponential convergence relies on $\sigma$ being "sufficiently large." In practical applications, excessive regularization can limit the expressive power of the neural network, potentially leading to "blurry" or suboptimal policies.
  • Architectural Constraints: The current analysis is restricted to one-hidden-layer networks. Extending these proofs to deep, multi-layer architectures remains a significant challenge in the field.
  • Idealized Gradients: The study assumes access to exact gradients. In real-world RL, agents must work with noisy estimates derived from environment samples. Bridging the gap between this theoretical gradient flow and actual "sample complexity" is a necessary next step.

Conclusion

By combining mean-field analysis with double entropy regularization, this paper provides the first global convergence guarantees for policy gradient methods using neural network approximations in continuous spaces. It transforms a traditionally mysterious and non-convex problem into a structured, convergent process, offering a vital theoretical anchor for the next generation of RL algorithms.

Summary updated

BibTeX
@inproceedings{kerimkulov2022convergence,
  author = {Bekzhan Kerimkulov and James-Michael Leahy and David Siska and Lukasz Szpruch},
  title = {{Convergence of policy gradient for entropy regularized MDPs with neural network approximation in the mean-field regime}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
}
Figure