A Max-Min Entropy Framework for Reinforcement Learning

Seungyul Han, Youngchul Sung

TL;DR

Maximum entropy reinforcement learning (like Soft Actor-Critic) is designed to boost exploration, but in practice, it often gets trapped in a positive feedback loop that narrows its focus. This paper introduces a Max-Min Entropy (MME) framework that actively seeks out low-entropy states and maximizes their entropy, breaking this loop. By disentangling exploration from exploitation, the proposed method achieves state-of-the-art performance on notoriously difficult sparse-reward and high-dimensional continuous control tasks.

What Problem Does This Paper Address?

In reinforcement learning (RL), exploration is a massive challenge, especially in environments where rewards are sparse or delayed. To encourage agents to try new things, researchers often use Maximum Entropy RL—most famously implemented in the Soft Actor-Critic (SAC) algorithm. The idea is to add an "entropy bonus" to the reward, encouraging the policy to remain as random as possible while still achieving high returns.

However, the authors identify a critical flaw in how SAC operates in practice. Because SAC uses off-policy learning with a replay buffer and neural network approximations, it suffers from a "saturation" problem. The Q-function in SAC estimates the sum of future entropy. As the agent visits certain states, those states are stored in the replay buffer and their Q-values rise. The policy then greedily prefers these high Q-value states, causing the policy distribution to skew toward specific actions. Ironically, this skew lowers the policy's entropy, trapping the agent in a narrow exploration radius.

SAC Exploration Paradox

The authors show that in a pure-exploration maze with no rewards, SAC eventually explores fewer new states than a completely random, uniform policy.

Methods

To fix this paradox, the authors propose the Max-Min Entropy (MME) framework.

Instead of a "max-max" approach (maximizing the policy entropy of states that already have high future entropy), MME flips the script. It defines a "reversed" soft Q-function that subtracts policy entropy from the reward. This trains the Q-function to identify and estimate states with low entropy. The policy is then updated to visit these low-entropy states, while simultaneously maximizing the policy's entropy once it gets there. This max-min dynamic forces the agent to fairly distribute its exploration across the state space.

Furthermore, the authors introduce Disentangled MME (DE-MME) for environments that actually have external rewards. Mixing external rewards and negative entropy into a single Q-function muddles the learning signal. To solve this, DE-MME splits the architecture:

  1. An exploration Q-function ($Q_{R,E}$) that tracks future negative entropy.
  2. A reward Q-function ($Q_{R,R}$) that tracks standard future returns.
  3. Two policies: A pure exploration policy ($\pi_E$) that seeks low-entropy states, and a target policy ($\pi_T$) that balances maximizing the actual reward with the exploration signal from $\pi_E$.

Datasets

The authors evaluate their methods on several simulated continuous control environments:

  • Pure Exploration: A custom 100x100 continuous 4-room maze (no rewards).
  • SparseMujoco: Modified OpenAI Gym Mujoco tasks (Hopper, HalfCheetah, Walker2d, Ant) where the agent only receives a reward of 1 after crossing a specific distance threshold.
  • DelayedMujoco: Mujoco tasks where rewards are accumulated and only given to the agent once every $D$ time steps.
  • High-Dimensional Control: Standard dense-reward Humanoid and Humanoid-Standup tasks.

Key Insights and Ideas

  • The SAC Paradox: Maximizing future entropy using off-policy experience replay creates a positive feedback loop that actually hinders exploration. Frequently visited states get higher Q-values, biasing the policy toward them and shrinking the exploration radius.
  • The Max-Min Principle: By treating exploration as a max-min optimization problem (seeking the minimum entropy states and maximizing them), the agent explores the state space much more fairly and uniformly, avoiding local saturation.
  • Decoupling Entropy Coefficients: MME separates the entropy coefficient used in the Q-function ($\alpha_Q$) from the one used in the policy update ($\alpha_\pi$). This allows fine-grained control over the trade-off between seeking novel states and taking diverse actions.
  • Exploitation and Exploration Must Be Disentangled: When external rewards are introduced, trying to do max-min entropy and reward maximization in a single Q-function fails. Explicitly separating the reward Q-value from the entropy Q-value (DE-MME) is necessary to get the best of both worlds.

Key Results

  • Pure Exploration: In the 4-room maze, SAC's exploration saturates at roughly 2,500 visited states. MME continues exploring throughout training, reaching over 4,500 states, easily beating both SAC and a uniform random policy.
  • Sparse Rewards: DE-MME achieves massive performance gains over baselines. On SparseWalker2d, DE-MME achieves a max average return of 886.60, compared to SAC's 278.50 and the popular Random Network Distillation (RND) method's 750.90.
  • Delayed Rewards: On Delayed HalfCheetah, DE-MME scores 8451.20, more than doubling SAC's score of 4080.67.
  • High-Dimensional Tasks: On Humanoid-Standup, DE-MME scores ~250k compared to SAC's ~138k, proving the framework scales well to complex, dense-reward environments.

Limitations and Future Work

Limitations: The authors do not heavily dwell on algorithmic limitations in the text. However, it is apparent that DE-MME requires maintaining multiple Q-functions (one for reward, one for entropy) and multiple policies (exploration and target), which increases computational overhead, memory usage, and architectural complexity compared to vanilla SAC.

Future Work: The authors briefly suggest that these enhanced exploration techniques could be applied to sensitive real-world control tasks (like drone control), where safe but wide exploration is necessary to find optimal policies without getting stuck in local optima.

Summary updated

Abstract

In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms.
BibTeX
@inproceedings{han2021max,
  author = {Seungyul Han and Youngchul Sung},
  title = {{A Max-Min Entropy Framework for Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=jmV8orBxgh},
}
Figure