Reinforcement learning with experience replay and adaptation of action dispersion

Pawel Wawrzynski, Wojciech Masarczyk, Mateusz OstaszewskiarXiv 2022 · DOI · OpenReview

Introduction

In Reinforcement Learning (RL), agents must constantly navigate the exploration-exploitation trade-off: deciding whether to exploit known high-reward paths or explore new actions to discover potentially better strategies. Traditional methods, such as Soft Actor-Critic (SAC), rely on manually tuned hyperparameters like entropy bonuses to manage this "action dispersion." However, these weights often require extensive task-specific trial-and-error. This paper introduces ACERAX (Actor-Critic with Experience Replay and Adaptive eXploration), an algorithm designed to automatically tune exploration levels by maintaining policy proximity to past successful actions.

The ACERAX Method

The core philosophy of ACERAX is that exploration should be "future-proof." Instead of exploring for the sake of randomness, the agent ensures its current policy remains wide enough to encompass the actions and distribution modes that were successful in the past. This approach ensures that the data collected early in training remains useful for evaluating the more refined policies the agent develops later.

The ACERAX principle: ensuring the current policy (1) maintains likelihood for past actions ($a_i$) and modes ($m_i$) to prevent premature convergence.

ACERAX implements this through a dedicated, small neural network called the $\bar{\eta}$ Network, which predicts the standard deviation of actions based on the current state. The algorithm utilizes two primary optimization rules:

  1. R1: Ensuring the mode (peak) of the distribution that generated a past experience remains likely under the current policy.
  2. R2: Ensuring the actual past action taken remains likely under the current policy.

By using experience replay with "softly truncated density ratios," ACERAX accounts for policy shifts over time. This allows the action dispersion to decay naturally as the agent masters a task and the distance between current and past policies shrinks, removing the need for a manual decay schedule.

Experimental Evaluation

The researchers evaluated ACERAX on four high-dimensional continuous control benchmarks from the PyBullet physics simulator: Ant, HalfCheetah, Hopper, and Walker2D. These environments are standard "stress tests" for RL due to their complex robotic dynamics and high-dimensional action spaces.

Key Results and Robustness

ACERAX demonstrated performance comparable to, and in most cases slightly exceeding, state-of-the-art baselines like SAC and PPO. A significant finding was the algorithm's ability to converge to optimal dispersion values (approximately 0.4) without manual intervention—the same value identified as optimal through exhaustive grid searches for baseline models.

Performance curves on the Ant benchmark demonstrating the algorithm's robustness across various values of the $\alpha$ hyperparameter.

Furthermore, the algorithm proved robust to the $\alpha$ coefficient, which weights the importance of past actions versus past modes. As shown in the performance plots for the Ant benchmark, while there is some variation at very low values, the agent consistently learns effective policies across a wide range of $\alpha$ settings. This suggests that ACERAX is significantly easier to deploy across different problem types than entropy-based methods that require precise tuning.

Limitations and Future Work

While ACERAX offers a promising "set-it-and-forget-it" approach, some challenges remain. The $\alpha$ parameter, though robust, still exists within the loss function. Additionally, the $\bar{\eta}$ network must be kept small (roughly 10x fewer neurons than the main policy) to avoid overfitting and premature collapse of randomness. Determining the ideal architecture for this network in even more complex domains, such as vision-based RL, remains an open question for future research.

Conclusion

ACERAX represents a step toward autonomous reinforcement learning by removing the need for manual exploration tuning. By linking exploration to the utility of past data for future policy evaluation, it provides a principled way for agents to decide how much to explore, leading to more robust and autonomous RL systems.

Summary updated

Abstract

Effective reinforcement learning requires a proper balance of exploration and exploitation defined by the dispersion of action distribution. However, this balance depends on the task, the current stage of the learning process, and the current environment state. Existing methods that designate the action distribution dispersion require problem-dependent hyperparameters. In this paper, we propose to automatically designate the action distribution dispersion using the following principle: This distribution should have sufficient dispersion to enable the evaluation of future policies. To that end, the dispersion should be tuned to assure a sufficiently high probability (densities) of the actions in the replay buffer and the modes of the distributions that generated them, yet this dispersion should not be higher. This way, a policy can be effectively evaluated based on the actions in the buffer, but exploratory randomness in actions decreases when this policy converges. The above principle is verified here on challenging benchmarks Ant, HalfCheetah, Hopper, and Walker2D, with good results. Our method makes the action standard deviations converge to values similar to those resulting from trial-and-error optimization.
BibTeX
@misc{wawrzynski2022reinforcement,
  author = {Pawel Wawrzynski and Wojciech Masarczyk and Mateusz Ostaszewski},
  title = {{Reinforcement learning with experience replay and adaptation of action dispersion}},
  year = {2022},
  eprint = {2208.00156},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.00156},
}
Figure