Abstract
We aim to develop off-policy DRL algorithms that not only exceed state-of-the-art performance but are also simple and minimalistic. For standard continuous control benchmarks, Soft Actor-Critic (SAC), which employs entropy maximization, currently provides state-of-the-art performance. We first demonstrate that the entropy term in SAC addresses action saturation due to the bounded nature of the action spaces, with this insight, we propose a streamlined algorithm with a simple normalization scheme or with inverted gradients. We show that both approaches can match SAC's sample efficiency performance without the need of entropy maximization, we then propose a simple non-uniform sampling method for selecting transitions from the replay buffer during training. Extensive experimental results demonstrate that our proposed sampling scheme leads to state of the art sample efficiency on challenging continuous control tasks. We combine all of our findings into one simple algorithm, which we call Streamlined Off Policy with Emphasizing Recent Experience, for which we provide robust public-domain code.
Introduction: The Complexity of Off-Policy Learning
In Deep Reinforcement Learning (DRL), "off-policy" algorithms are highly valued for their ability to reuse past data, making them significantly more sample-efficient than on-policy methods. Currently, Soft Actor-Critic (SAC) is considered the gold standard for continuous control tasks. SAC's success is largely attributed to "maximum entropy RL," a framework that adds an entropy bonus to the objective function to encourage exploration by rewarding the agent for being "random."
However, this paper investigates whether SAC's complexity is strictly necessary. The authors identify a specific failure mode in simpler off-policy algorithms termed the squashing exploration problem. In continuous control, actions are typically bounded (e.g., within [-1, 1]). To enforce these bounds, neural networks often use a tanh activation function. If the network outputs a very large value, the tanh function saturates, and any added exploration noise becomes negligible, effectively killing exploration.
Methods: Streamlining the Off-Policy Pipeline
The researchers propose two primary components to simplify off-policy DRL while maintaining or improving performance:
1. Output Normalization (SOP Algorithm)
Instead of relying on complex entropy maximization to prevent deterministic policies, the authors introduce the Streamlined Off-Policy (SOP) algorithm. SOP employs a simple output normalization scheme: it monitors the magnitudes of the policy network's outputs. If the average magnitude across action dimensions exceeds 1.0, the outputs are scaled down. This keeps the pre-activation values within the sensitive range of the tanh function, ensuring that exploration noise remains effective.
2. Emphasizing Recent Experience (ERE)
Standard off-policy algorithms sample transitions uniformly from a replay buffer. While Prioritized Experience Replay (PER) is a popular alternative, it is computationally intensive. The authors propose Emphasizing Recent Experience (ERE), a minimalistic non-uniform sampling scheme. During training, ERE gradually shrinks the sampling window to focus more on recent data. This allows the agent to prioritize new information while still retaining a degree of historical knowledge, avoiding the complexity of PER's sum-tree structures.
Experimental Setup
The proposed methods were evaluated using the MuJoCo (Multi-Joint dynamics with Contact) physics engine, the standard benchmark for continuous control. The study focused on challenging environments including Humanoid-v2, which features 17 action dimensions and is notoriously difficult to stabilize, as well as Ant-v2, Walker2d-v2, Hopper-v2, and HalfCheetah-v2.
The "Squashing Exploration" Insight
A key contribution of this work is the empirical demonstration that the primary role of entropy maximization in SAC is preventing action saturation. As shown in the detailed analysis below, when entropy is removed from SAC, the model's internal outputs ($\mu_k$) explode to magnitudes as high as $10^4$. This causes the resulting actions ($a_k$) to saturate at the boundaries, rendering exploration noise useless.
By replacing the theoretical complexity of maximum entropy with a simple normalization step, the SOP algorithm achieves comparable results, suggesting that many theoretical breakthroughs may actually be addressing practical numerical stability issues.
Key Results
- SOP Matches SAC: The streamlined SOP algorithm with output normalization matched SAC's performance across all benchmarks, proving that entropy maximization is not the only path to state-of-the-art results.
- SOP + ERE = New SOTA: Combining SOP with the ERE sampling scheme resulted in a new state-of-the-art for sample efficiency. On the challenging Humanoid-v2 task, the combined approach improved performance by 24% over the SAC baseline at 1 million training samples.
- Ablation Studies: Removing normalization from SOP caused performance on Humanoid to collapse, confirming that managing action saturation is the critical factor for success in these environments.
Conclusion and Future Work
This research demonstrates that high-performance off-policy DRL can be achieved through simpler, more intuitive mechanisms than previously thought. By addressing the "squashing exploration" problem directly through normalization and utilizing a recency-biased sampling strategy, the authors provide a robust and streamlined alternative to complex state-of-the-art methods.
Future work includes evaluating these simplified approaches in discrete action spaces (e.g., Atari) and vision-based tasks to determine if the benefits of simplicity generalize beyond MuJoCo benchmarks. Additionally, while ERE is effective, the trade-offs of its hyperparameters in non-stationary environments warrant further investigation.
Summary updated