Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation

Qingling Zhu, Xiaoqiang Wu, Qiuzhen Lin, Wei-Neng ChenAAAI 2024 · DOI

Abstract

Deep Reinforcement Learning (DRL) often suffers from premature convergence to sub-optimal strategies. While Evolutionary Reinforcement Learning (ERL) addresses this by maintaining a population of diverse actors, existing methods typically rely on a single shared critic, which can become a bottleneck. We present Two-Stage Evolutionary Reinforcement Learning (TERL), a framework that evolves complete actor-critic pairs. By transitioning from a multi-agent exploration phase to a focused exploitation phase, TERL effectively overcomes local optima and sparse reward challenges in complex continuous control tasks.

Introduction

A persistent challenge in Deep Reinforcement Learning is the tendency for agents to get stuck in local optima, a phenomenon known as premature convergence. Evolutionary Reinforcement Learning (ERL) was introduced to mitigate this by maintaining a population of diverse actor networks. However, most ERL implementations utilize a single, shared critic to evaluate the entire population. If this critic falls into a local optimum, it provides flawed guidance, causing the entire population to stall. TERL addresses this by ensuring every individual in the population is a complete agent with its own actor and critic network.

Methodology

TERL operates through two distinct training phases designed to balance the search for global optima with the fine-tuning of high-performing strategies.

1. Exploration Stage

In the initial phase, every individual in the population learns independently. Each agent maintains its own actor and critic, allowing for diverse "perspectives" on the reward landscape. To maintain computational efficiency and facilitate knowledge transfer, agents share a common replay buffer and utilize Particle Swarm Optimization (PSO). PSO periodically nudges actor networks toward the parameters of the current global best individual (gBest), ensuring the population benefits from collective discoveries without losing individual diversity.

2. Exploitation Stage

Once a predefined "exploration ratio" of training time is reached, the algorithm shifts focus. The best-performing individual is identified, and the majority of RL gradient-update resources are allocated to it. While the remaining individuals stop updating their critics, they continue to provide diverse experiences through PSO-based updates to their actors, serving as "scouts" for the primary agent.

The TERL framework transitions from independent agent learning in the exploration stage to focused resource allocation in the exploitation stage.

Experimental Results

The researchers evaluated TERL on standard continuous control benchmarks, including MuJoCo physics simulations (e.g., HalfCheetah, Ant, Walker2d) and classic control tasks (e.g., BipedalWalker). TERL was compared against state-of-the-art ERL methods like PDERL and ERL-Re², as well as standard RL algorithms like TD3 and PPO.

TERL outperformed baselines in 7 out of 9 tested environments. Notably, in environments prone to local optima like Hopper and Walker2d, TERL achieved significantly higher final scores. While the overhead of training multiple critics leads to a slower start, the algorithm demonstrates a much steeper learning curve during the exploitation phase, eventually surpassing competing methods.

Performance curves show TERL (red) achieving superior final performance and faster convergence in the later stages of training compared to baselines.

Discussion and Conclusion

By evolving complete actor-critic pairs, TERL prevents the "blind leading the blind" scenario common in single-critic ERL methods. The strategic shift from broad exploration to targeted exploitation allows the framework to find superior strategies in complex environments.

Despite its success, the computational cost of maintaining multiple critics remains a factor. Future work will focus on adaptive mechanisms for switching between stages based on performance plateaus and integrating advanced selection methods like Novelty Search to further enhance population diversity in sparse-reward scenarios.

Summary updated

BibTeX
@inproceedings{zhu2024two,
  author = {Qingling Zhu and Xiaoqiang Wu and Qiuzhen Lin and Wei-Neng Chen},
  title = {{Two-Stage Evolutionary Reinforcement Learning for Enhancing Exploration and Exploitation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i18.30079},
}
Figure