Abstract
Deep Reinforcement Learning (DRL) often suffers from premature convergence to sub-optimal strategies. While Evolutionary Reinforcement Learning (ERL) addresses this by maintaining a population of diverse actors, existing methods typically rely on a single shared critic, which can become a bottleneck. We present Two-Stage Evolutionary Reinforcement Learning (TERL), a framework that evolves complete actor-critic pairs. By transitioning from a multi-agent exploration phase to a focused exploitation phase, TERL effectively overcomes local optima and sparse reward challenges in complex continuous control tasks.
Introduction
A persistent challenge in Deep Reinforcement Learning is the tendency for agents to get stuck in local optima, a phenomenon known as premature convergence. Evolutionary Reinforcement Learning (ERL) was introduced to mitigate this by maintaining a population of diverse actor networks. However, most ERL implementations utilize a single, shared critic to evaluate the entire population. If this critic falls into a local optimum, it provides flawed guidance, causing the entire population to stall. TERL addresses this by ensuring every individual in the population is a complete agent with its own actor and critic network.
Methodology
TERL operates through two distinct training phases designed to balance the search for global optima with the fine-tuning of high-performing strategies.
1. Exploration Stage
In the initial phase, every individual in the population learns independently. Each agent maintains its own actor and critic, allowing for diverse "perspectives" on the reward landscape. To maintain computational efficiency and facilitate knowledge transfer, agents share a common replay buffer and utilize Particle Swarm Optimization (PSO). PSO periodically nudges actor networks toward the parameters of the current global best individual (gBest), ensuring the population benefits from collective discoveries without losing individual diversity.
2. Exploitation Stage
Once a predefined "exploration ratio" of training time is reached, the algorithm shifts focus. The best-performing individual is identified, and the majority of RL gradient-update resources are allocated to it. While the remaining individuals stop updating their critics, they continue to provide diverse experiences through PSO-based updates to their actors, serving as "scouts" for the primary agent.
Experimental Results
The researchers evaluated TERL on standard continuous control benchmarks, including MuJoCo physics simulations (e.g., HalfCheetah, Ant, Walker2d) and classic control tasks (e.g., BipedalWalker). TERL was compared against state-of-the-art ERL methods like PDERL and ERL-Re², as well as standard RL algorithms like TD3 and PPO.
TERL outperformed baselines in 7 out of 9 tested environments. Notably, in environments prone to local optima like Hopper and Walker2d, TERL achieved significantly higher final scores. While the overhead of training multiple critics leads to a slower start, the algorithm demonstrates a much steeper learning curve during the exploitation phase, eventually surpassing competing methods.
Discussion and Conclusion
By evolving complete actor-critic pairs, TERL prevents the "blind leading the blind" scenario common in single-critic ERL methods. The strategic shift from broad exploration to targeted exploitation allows the framework to find superior strategies in complex environments.
Despite its success, the computational cost of maintaining multiple critics remains a factor. Future work will focus on adaptive mechanisms for switching between stages based on performance plateaus and integrating advanced selection methods like Novelty Search to further enhance population diversity in sparse-reward scenarios.
Summary updated