Maximum Entropy Softmax Policy Gradient via Entropy Advantage Estimation

Jean Seong Bjorn Choe, Jong-Kook KimIJCAI 2025 · OpenReview

TL;DR

This paper introduces Entropy Advantage Policy Optimisation (EAPO), a novel method that simplifies the application of Maximum Entropy Reinforcement Learning (MaxEnt RL) to standard policy gradient algorithms. EAPO achieves improved performance, generalization, and exploration by separately estimating task and entropy objectives, offering a more stable and effective MaxEnt RL approach.

What Problem Does This Paper Address?

Maximum Entropy Reinforcement Learning (MaxEnt RL) is a powerful framework that encourages policies to explore more broadly by adding an entropy bonus to the objective function. Theoretically, this leads to better policy optimization and stability. However, applying MaxEnt RL directly to common policy gradient methods like Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO) has been surprisingly underexplored. The authors hypothesize this is due to the practical difficulty of managing the entropy reward, which can lead to issues like reward inflation and unstable learning, especially in episodic tasks. Existing methods often struggle with this, leading to suboptimal policies or requiring complex modifications.

Methods

The core contribution of this paper is the Entropy Advantage Policy Optimisation (EAPO) algorithm. EAPO tackles the challenge of integrating MaxEnt RL by:

  • Separating Task and Entropy Objectives: Instead of naively adding an entropy reward to the existing task reward, EAPO estimates the task advantage and the entropy advantage independently.
  • Entropy Advantage Estimation: It uses a dedicated "entropy critic" network to estimate the trajectory entropy. This entropy advantage is then calculated using Generalised Advantage Estimation (GAE), similar to how the standard task advantage is computed, but using negative log probabilities instead of rewards.
  • Soft Advantage Function: EAPO combines the estimated task advantage and entropy advantage to form a "soft advantage" function. This soft advantage is then used to update the policy.
  • Leveraging Existing Frameworks: EAPO is designed to be easily integrated into existing policy gradient algorithms like PPO and TRPO by simply replacing the standard advantage estimate with the proposed soft advantage. This minimal modification makes it practical to implement.
  • Dedicated Discount Factors: EAPO uses separate discount factors for the task reward ($\gamma_v$) and the entropy reward ($\gamma_H$). This allows for finer control over the effective horizon of each objective, helping to mitigate issues like reward inflation.

Datasets

The paper evaluates EAPO on several benchmark environments:

  • Procgen Benchmark Environments: 16 procedurally generated environments used to test generalization capabilities.
  • MuJoCo Continuous Control Tasks: 4 discretised continuous control tasks (Ant-v4, Walker2d-v4, HalfCheetah-v4, Humanoid-v4) to evaluate performance on continuing control tasks.
  • MiniGrid-DoorKey-8x8 Environment: A challenging environment used to assess exploration performance.

Key Insights and Ideas

  • The "Entropy Reward" is Tricky: The paper highlights that simply adding an entropy bonus to the reward signal can lead to problematic behaviors like reward inflation, making it difficult for agents to learn optimal policies, especially in episodic settings.
  • Separation is Key: EAPO's core idea of separately estimating task and entropy objectives is crucial. This allows for more controlled learning and avoids the pitfalls of a combined, potentially unstable, reward signal.
  • Entropy Advantage as a Bridge: By framing the entropy objective as an "entropy advantage" and using GAE, EAPO effectively bridges the gap between theoretical MaxEnt RL and practical policy gradient methods.
  • Controlling the Entropy Horizon: The use of a separate discount factor ($\gamma_H$) for the entropy objective is a simple yet effective way to manage its influence and prevent it from dominating the learning process.
  • MaxEnt RL for Better Generalization: The results suggest that MaxEnt policies, particularly those with higher entropy, exhibit improved robustness to distributional shifts, leading to better generalization performance.

Key Results

  • Improved Performance on Procgen: EAPO significantly outperformed baseline PPO agents on the Procgen benchmark, achieving a 29% higher mean normalized score and showing substantial improvements in specific environments like Plunder, BigFish, and Dodgeball.
  • Stable MaxEnt for Discrete Actions: EAPO demonstrated superior stability and performance compared to SD-SAC (a discrete SAC variant) on Procgen environments.
  • Competitive on MuJoCo: EAPO, when extended to PPO and TRPO, achieved competitive performance on continuous control tasks, often outperforming baselines with naive entropy augmentation.
  • Enhanced Exploration: In the challenging MiniGrid-DoorKey-8x8 environment, EAPO successfully solved the task in significantly fewer frames and more consistently than baseline PPO.
  • Generalization Benefits: Higher entropy policies trained with EAPO showed better performance in testing scenarios compared to training, indicating improved robustness to unseen variations.

Limitations and Future Work

  • Sensitivity to Hyperparameters: The paper notes that EAPO, like many RL algorithms, can be sensitive to hyperparameters, particularly $\gamma_H$ and $\lambda_H$. While EAPO's improvement in success rate on MiniGrid-DoorKey-8x8 contradicts the theory that entropy regularization might not effectively mitigate epistemic uncertainty, the authors suggest this might be due to the on-policy nature of their entropy estimation, which could underestimate entropy for previously visited trajectories, implicitly encouraging exploration.
  • Continuing Tasks: While EAPO shows stable performance, its efficiency gains on continuing tasks might be less pronounced compared to episodic tasks.
  • Broader Adoption: The authors anticipate that EAPO's simplicity and effectiveness will encourage wider adoption of MaxEnt RL and inference-based methods in emerging areas like competitive RL and robust RL. Future work could explore the combination of KL and entropy regularization more deeply.

Summary updated

Abstract

Entropy Regularisation is a widely adopted technique that enhances policy optimisation performance and stability. Maximum entropy reinforcement learning (MaxEnt RL) regularises policy evaluation by augmenting the objective with an entropy term, showing theoretical benefits in policy optimisation. However, its practical application in straightforward direct policy gradient settings remains surprisingly underexplored. We hypothesise that this is due to the difficulty of managing the entropy reward in practice. This paper proposes Entropy Advantage Policy Optimisation (EAPO), a simple method that facilitates MaxEnt RL implementation by separately estimating task and entropy objectives. Our empirical evaluations demonstrate that extending Proximal Policy Optimisation (PPO) and Trust Region Policy Optimisation (TRPO) within the MaxEnt framework improves optimisation performance, generalisation, and exploration in various environments. Moreover, our method provides a stable and performant MaxEnt RL algorithm for discrete action spaces.
BibTeX
@inproceedings{choe2025maximum,
  author = {Jean Seong Bjorn Choe and Jong-Kook Kim},
  title = {{Maximum Entropy Softmax Policy Gradient via Entropy Advantage Estimation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=5tOTr5AQwJ},
}
Figure