This survey paper, authored by Olivier Sigaud, provides a comprehensive map of the rapidly growing landscape where Deep Reinforcement Learning (DRL) meets Deep Neuroevolution (DNE). By analyzing 45 algorithms developed since 2017, the paper establishes a unified framework for understanding how population-based search and gradient-based learning can work together to solve complex policy search problems.
TL;DR
This paper surveys the "marriage" of Deep Reinforcement Learning and Neuroevolution, categorizing how gradient-based optimization and population-based selection can be combined. It highlights how these hybrid approaches achieve superior performance, robustness, and diversity compared to using either method in isolation.
What Problem Does This Paper Address?
Deep Reinforcement Learning (DRL) has seen massive success, but it remains "brittle." DRL relies on estimating gradients from limited samples, which can lead to high variance and "blind" updates that occasionally destroy a policy's performance. On the other hand, Evolutionary Algorithms (EA) are incredibly robust and excellent at global exploration because they evaluate policies holistically, but they are notoriously sample-inefficient.
The core challenge addressed here is: How can we combine the sample efficiency of DRL with the robustness and exploratory power of Evolution? The paper seeks to organize the "zoo" of recent hybrid algorithms into a coherent taxonomy based on how these two paradigms interact.
Methods
The survey organizes the literature into four primary categories of combination:
- Evolution of Policies for Performance: These methods run an evolutionary loop and a DRL loop in parallel. A key technique is Actor Injection (seen in ERL), where a DRL-trained agent is periodically added to an evolutionary population to "boost" its performance. Other methods, like CEM-RL, apply DRL gradient steps directly to members of the evolutionary population to accelerate their improvement.
- Evolution of Actions: Instead of evolving entire policies, some algorithms use evolutionary strategies (like the Cross-Entropy Method or PSO) to find the best action to take in a specific state. This is particularly useful in continuous action spaces where finding the maximum of a Q-function is computationally expensive.
- Evolution for Diversity: These methods (Novelty Search and Quality-Diversity) focus on maintaining a population of agents that are not just high-performing, but also behaviorally different. This prevents the system from getting stuck in local optima and is crucial for tasks with sparse rewards.
- Evolution of "Something Else": Evolution is used to optimize components that gradients cannot easily reach, such as hyperparameters (Population Based Training), neural network architectures, reward functions, or even the physical morphology of a robot.
Datasets
As this is a survey paper, it does not introduce a new dataset or benchmark. Instead, it references the standard benchmarks used across the 45 surveyed papers, which primarily include:
- MuJoCo / PyBullet: For continuous control and locomotion tasks.
- Atari 2600: For discrete action reinforcement learning.
- Robotic Manipulation Tasks: Used in papers like QT-Opt.
- Hard Exploration Environments: Such as "Monteuma’s Revenge" or specialized "Deceptive" landscapes.
Key Insights and Ideas
- Evolution as a Safety Filter: One of the most powerful "aha" moments is that Evolution can act as a filter for DRL. While DRL might suggest a gradient update that ruins a policy, the evolutionary selection process will simply discard that "failed" individual, ensuring the population only moves toward better performance.
- The Critic as a Bridge: In many hybrid designs, the DRL "Critic" (which estimates the value of states/actions) serves as a Surrogate Fitness function. This allows the evolutionary loop to evaluate thousands of "virtual" individuals without ever having to run a slow, expensive environment simulation.
- Structure vs. Parameters: Evolution is uniquely suited to exploring discrete structures (like network topology or robot limbs), while DRL is best at fine-tuning the continuous weights within those structures.
- Diversity is a First-Class Citizen: Combining DRL with Quality-Diversity (QD) algorithms allows for "archives" of skills. Instead of one master agent, you get a library of agents that solve the problem in different ways, leading to extreme robustness.
Key Results
While the paper focuses on mechanisms rather than raw numbers, it summarizes several high-level quantitative trends:
- Sample Efficiency: Hybrid methods like CERL and PDERL consistently show faster convergence than pure EA and more stability than pure DRL (like DDPG or TD3).
- Global Optima: In "deceptive" environments where a simple gradient would lead to a dead end, hybrid methods using Novelty Search (like NS-RL) are often the only ones capable of finding the solution.
- Scalability: Algorithms like QT-Opt demonstrate that using evolutionary action selection can scale to massive, real-world robotic grasping tasks involving thousands of hours of data.
Limitations and Future Work
The author notes that the field currently lacks a unified benchmark. Because different researchers use different environments and evaluation metrics, it is difficult to perform a true "head-to-head" comparison of all 45 algorithms.
Future Directions: * "Combining the Combinations": The next frontier is creating a single architecture that uses all these tricks at once: surrogate fitness, actor injection, hyperparameter tuning, and diversity seeking. * Asynchrony: Most current frameworks are synchronous, but the author suggests that moving toward fully asynchronous, "population-scale" learning will be necessary to leverage modern massive computing clusters. * Policy Distillation: Finding better ways to "compress" the knowledge of a large evolutionary population into a single, efficient DRL agent.
Summary updated