Introduction
In Reinforcement Learning (RL), agents must constantly navigate the exploration-exploitation trade-off: deciding whether to exploit known high-reward paths or explore new actions to discover potentially better strategies. Traditional methods, such as Soft Actor-Critic (SAC), rely on manually tuned hyperparameters like entropy bonuses to manage this "action dispersion." However, these weights often require extensive task-specific trial-and-error. This paper introduces ACERAX (Actor-Critic with Experience Replay and Adaptive eXploration), an algorithm designed to automatically tune exploration levels by maintaining policy proximity to past successful actions.
The ACERAX Method
The core philosophy of ACERAX is that exploration should be "future-proof." Instead of exploring for the sake of randomness, the agent ensures its current policy remains wide enough to encompass the actions and distribution modes that were successful in the past. This approach ensures that the data collected early in training remains useful for evaluating the more refined policies the agent develops later.
ACERAX implements this through a dedicated, small neural network called the $\bar{\eta}$ Network, which predicts the standard deviation of actions based on the current state. The algorithm utilizes two primary optimization rules:
- R1: Ensuring the mode (peak) of the distribution that generated a past experience remains likely under the current policy.
- R2: Ensuring the actual past action taken remains likely under the current policy.
By using experience replay with "softly truncated density ratios," ACERAX accounts for policy shifts over time. This allows the action dispersion to decay naturally as the agent masters a task and the distance between current and past policies shrinks, removing the need for a manual decay schedule.
Experimental Evaluation
The researchers evaluated ACERAX on four high-dimensional continuous control benchmarks from the PyBullet physics simulator: Ant, HalfCheetah, Hopper, and Walker2D. These environments are standard "stress tests" for RL due to their complex robotic dynamics and high-dimensional action spaces.
Key Results and Robustness
ACERAX demonstrated performance comparable to, and in most cases slightly exceeding, state-of-the-art baselines like SAC and PPO. A significant finding was the algorithm's ability to converge to optimal dispersion values (approximately 0.4) without manual intervention—the same value identified as optimal through exhaustive grid searches for baseline models.
Furthermore, the algorithm proved robust to the $\alpha$ coefficient, which weights the importance of past actions versus past modes. As shown in the performance plots for the Ant benchmark, while there is some variation at very low values, the agent consistently learns effective policies across a wide range of $\alpha$ settings. This suggests that ACERAX is significantly easier to deploy across different problem types than entropy-based methods that require precise tuning.
Limitations and Future Work
While ACERAX offers a promising "set-it-and-forget-it" approach, some challenges remain. The $\alpha$ parameter, though robust, still exists within the loss function. Additionally, the $\bar{\eta}$ network must be kept small (roughly 10x fewer neurons than the main policy) to avoid overfitting and premature collapse of randomness. Determining the ideal architecture for this network in even more complex domains, such as vision-based RL, remains an open question for future research.
Conclusion
ACERAX represents a step toward autonomous reinforcement learning by removing the need for manual exploration tuning. By linking exploration to the utility of past data for future policy evaluation, it provides a principled way for agents to decide how much to explore, leading to more robust and autonomous RL systems.
Summary updated