Abstract
In this paper, we propose a max-min entropy framework for reinforcement learning (RL) to overcome the limitation of the soft actor-critic (SAC) algorithm implementing the maximum entropy RL in model-free sample-based learning. Whereas the maximum entropy RL guides learning for policies to reach states with high entropy in the future, the proposed max-min entropy framework aims to learn to visit states with low entropy and maximize the entropy of these low-entropy states to promote better exploration. For general Markov decision processes (MDPs), an efficient algorithm is constructed under the proposed max-min entropy framework based on disentanglement of exploration and exploitation. Numerical results show that the proposed algorithm yields drastic performance improvement over the current state-of-the-art RL algorithms.
Introduction: The "Boredom" Problem in RL
In Reinforcement Learning (RL), exploration is the critical process of discovering new states to find optimal rewards. A dominant approach in recent years has been Maximum Entropy RL, most notably implemented via the Soft Actor-Critic (SAC) algorithm. The core philosophy is to maximize not just the expected reward, but also the "randomness" (entropy) of the agent's actions. Theoretically, this prevents premature convergence to sub-optimal policies.
However, this paper identifies a fundamental flaw in this paradigm: Exploration Saturation. In practice, SAC utilizes a neural network to estimate future entropy. When an agent frequently visits a specific region of the state space, the network's estimate of future entropy for that area increases. This creates a pathological positive feedback loop: the agent is incentivized to return to familiar states because it perceives them as leading to high entropy, even if its actual behavior in those states has become predictable and "solved." Consequently, the agent becomes "bored" and stops pushing into the unknown, resulting in a limited exploration radius.
The Max-Min Entropy (MME) Framework
To break this feedback loop, we introduce the Max-Min Entropy (MME) framework. Instead of seeking states with high future entropy, MME directs the agent toward states where its current policy has low entropy—areas where it is currently "certain" or stuck in a specific behavior. Once the agent reaches these low-entropy frontiers, the objective shifts to maximizing the entropy of actions taken there.
The Reversed Q-Function
Implementation of this framework involves a "reversed" soft Q-function ($Q_R$). In standard SAC, entropy is added to the reward signal. In MME, entropy is subtracted during the value estimation phase. This effectively penalizes states where the agent already exhibits high entropy and provides a relative incentive to move toward low-entropy regions. This "fairness" principle ensures that exploration resources are not over-invested in well-understood areas of the environment.
Disentangled MME (DE-MME)
In environments with external rewards, the exploration signal can often be overwhelmed by the exploitation objective. To address this, we propose Disentangled MME (DE-MME). This variant maintains two separate policies: one dedicated to pure exploration and another for the target task. This separation allows the agent to maintain high curiosity and robust exploration without sacrificing performance on the primary goal.
Experimental Evaluation
We evaluated the MME framework across several challenging benchmarks designed to test exploration capabilities:
- Continuous 4-Room Maze: A 2D environment used to measure pure exploration via state visitation counts without external rewards.
- Sparse and Delayed MuJoCo: Modified versions of standard tasks (e.g., Hopper, Ant, HalfCheetah) where rewards are either binary (sparse) or accumulated and delivered only at long intervals (delayed), creating difficult credit-assignment problems.
- High-Dimensional Humanoid: Complex tasks from the MuJoCo suite that are notoriously difficult for standard RL due to their high-dimensional state and action spaces.
Key Results
The MME and DE-MME algorithms consistently outperformed state-of-the-art baselines, including SAC, Random Network Distillation (RND), and Diversity Actor-Critic (DAC).
- Exploration Coverage: In the 4-room maze, MME visited significantly more unique states than SAC. As shown in the experimental data, SAC tends to saturate and cease exploration after approximately 300k steps, while MME continues to expand its coverage.
- Sparse Reward Performance: On the SparseHalfCheetah-v1 task, DE-MME achieved a maximum average return of 924.90, more than double the 386.90 achieved by standard SAC.
- Complex Control: In the high-dimensional Humanoid task, MME reached a score of 9080.54, representing a nearly 35% improvement over SAC’s 6760.81.
- Stability: MME demonstrated lower variance across random seeds, suggesting that the self-correcting nature of the max-min objective leads to more stable training compared to other exploration-heavy methods.
Conclusion
The Max-Min Entropy framework offers a novel solution to the exploration saturation problem in RL. By reversing the relationship between the Q-function and entropy, we transform a self-reinforcing loop of "familiarity" into a self-correcting mechanism that pushes agents toward the unknown. While DE-MME introduces additional computational complexity due to its dual-policy architecture, the drastic improvements in sparse-reward and high-dimensional environments suggest that targeting low-entropy states is a powerful principle for robust reinforcement learning.
Summary updated