Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Method

Duo Xu, Faramarz Fekri

TL;DR

This paper introduces "Hamiltonian Policy Optimization" (HPO), a novel approach that integrates Hamiltonian Monte Carlo (HMC) methods into actor-critic reinforcement learning (RL). By using HMC to evolve actions from a base policy, HPO aims to bridge the "amortization gap" and improve exploration, leading to more efficient and effective learning, especially in complex continuous control tasks and safe RL scenarios.

What Problem Does This Paper Address?

Actor-critic RL algorithms are powerful tools for solving complex control problems, but they often struggle with two key issues: the "amortization gap" and insufficient exploration. The amortization gap arises when the policy network, trained to directly output action distributions, fails to perfectly approximate the true optimal posterior distribution of actions. This leads to suboptimal policies and a gap in the RL objective. Insufficient exploration means the agent might not discover the best actions, especially in high-dimensional action spaces. The paper frames RL as a variational inference (VI) problem and proposes using HMC, a technique from VI, to address these limitations.

Methods

The core innovation is the integration of Hamiltonian dynamics (HD) into the actor-critic framework, creating a "Hamiltonian policy." Instead of directly sampling actions from a learned policy distribution, HPO uses HMC to "evolve" actions. This involves:

  • Hamiltonian Dynamics (HD): The paper leverages the principles of HD, which simulate the motion of a particle on a potential energy landscape (defined by the Q-function). This simulation is performed using a series of discrete steps called the "leapfrog operator."
  • Proposed Leapfrog Operator: To adapt to the dynamic nature of RL (where Q-functions change during training), the authors introduce a novel, gated leapfrog operator. This operator integrates gradient information explicitly and implicitly, aiming for better expressivity and numerical stability compared to standard HMC leapfrog methods.
  • Policy Optimization: The overall objective is to optimize the policy parameters by maximizing the Evidence Lower Bound (ELBO) from the VI perspective. The HMC steps are incorporated into this optimization process.
  • Safe RL Integration: For safe RL, HPO iteratively samples actions using HMC until a Lyapunov constraint is satisfied, effectively discarding potentially unsafe actions.

Datasets

The paper evaluates its method on standard continuous control benchmarks:

  • MuJoCo Suite: Including HalfCheetah-v2, Hopper-v2, Walker2d-v2, Ant-v2, and Humanoid-v2.
  • PyBullet Roboschool: Including Humanoid-PyBullet, Flagrun-PyBullet, and Flagrun-Harder-PyBullet.

No new datasets were introduced.

Key Insights and Ideas

  • Bridging the Amortization Gap with HMC: The paper convincingly argues that HMC can help fill the "amortization gap" by allowing the policy to explore and refine actions based on the Q-function's landscape, leading to a better approximation of the target posterior.
  • HMC for Enhanced Exploration: HMC's inherent randomness in momentum vectors and its ability to "jump" over local optima are shown to significantly improve exploration efficiency, guiding the agent towards regions with higher Q-values.
  • Novel Gated Leapfrog Operator: The proposed gated leapfrog operator is crucial for adapting HMC to the dynamic RL setting, where the target distribution (defined by Q-functions) changes. This is a key technical contribution that differentiates it from standard HMC applications.
  • Improved Performance in Safe RL: The integration of HMC in safe RL settings not only improves returns but also demonstrably reduces safety constraint violations by iteratively sampling actions until they meet safety criteria.

Key Results

  • Outperforms Baselines: SAC-HPO consistently outperforms standard Soft Actor-Critic (SAC) and SAC with normalizing flow (SAC-NF) across various continuous control tasks in terms of convergence rate and final performance.
  • Significant Improvement in Safe RL: In safe RL environments (Ant-v2 and HalfCheetah-v2), SAC-HPO significantly reduces safety violations while achieving higher rewards compared to SAC-Lagrangian.
  • Data Efficiency: The paper suggests that the approach is data-efficient, implying it can learn effective policies with fewer interactions.
  • Action Distribution Expressivity: Visualizations show that HMC-evolved actions exhibit more non-Gaussian shapes and are better aligned with high Q-value regions compared to actions from a standard Gaussian policy.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper notes that the performance can be sensitive to hyperparameter choices, particularly the variance of momentum vectors in exploration.
  • Computational Cost: While not explicitly detailed as a limitation, HMC typically involves multiple simulation steps (leapfrog), which can increase the computational cost per training step compared to simpler policy updates.
  • Future Work: The paper implicitly suggests future work in exploring more sophisticated HMC adaptations for RL and potentially applying the method to even more complex and challenging environments. The ablation studies and sensitivity analyses also provide insights for tuning and further development.

Summary updated

Abstract

The actor–critic reinforcement learning (RL) is widely used in various robotic control tasks. By viewing the actor–critic RL from the perspective of variational inference (VI), the policy network is trained to obtain the approximate posterior of actions, given the optimality criteria. However, in practice, the actor–critic RL may yield suboptimal policy estimates due to the amortization gap and insufficient exploration. In this work, inspired by the previous use of Hamiltonian Monte Carlo (HMC) in VI, we propose to integrate the policy network of actor–critic RL with HMC, which is termed as Hamiltonian policy. As such we propose to evolve actions from the base policy according to HMC, and our proposed method has many benefits. First, HMC can improve the policy distribution to better approximate the posterior and hence reduce the amortization gap. Second, HMC can also guide the exploration more to the regions of action spaces with higher Q values, enhancing the exploration efficiency. Further, instead of directly applying HMC into RL, we propose a new leapfrog operator to simulate the Hamiltonian dynamics. Finally, in safe RL problems, we find that the proposed method can not only improve the achieved return, but also reduce safety constraint violations by discarding potentially unsafe actions. With comprehensive empirical experiments on continuous control baselines, including MuJoCo and PyBullet Roboschool, we show that the proposed approach is a data-efficient and easy-to-implement improvement over previous actor–critic methods.
BibTeX
@article{xu2023improving,
  author = {Duo Xu and Faramarz Fekri},
  title = {{Improving Actor-Critic Reinforcement Learning via Hamiltonian Monte Carlo Method}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=kjNxXSBX4r},
}
Figure