Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey

Olivier Sigaud

Abstract

Deep neuroevolution and deep Reinforcement Learning have received a lot of attention in the last years. Some works have compared them, highlighting theirs pros and cons, but an emerging trend consists in combining them so as to benefit from the best of both worlds. In this paper, we provide a survey of this emerging trend by organizing the literature into related groups of works and casting all the existing combinations in each group into a generic framework. We systematically cover all easily available papers irrespective of their publication status, focusing on the combination mechanisms rather than on the experimental results. In total, we cover 45 algorithms more recent than 2017. We hope this effort will favor the growth of the domain by facilitating the understanding of the relationships between the methods, leading to deeper analyses, outlining missing useful comparisons and suggesting new combinations of mechanisms.

Introduction

Deep Reinforcement Learning (DRL) has achieved remarkable success in complex tasks, yet it remains notoriously "brittle." Because DRL relies on estimating gradients from limited samples, it is prone to high variance and updates that can inadvertently degrade policy performance. Conversely, Evolutionary Algorithms (EA) offer robustness and global exploration capabilities by evaluating policies holistically, though they are often sample-inefficient.

This survey addresses the core challenge of merging these two paradigms: how can we combine the sample efficiency of DRL with the exploratory power of Evolution? By analyzing 45 algorithms developed since 2017, this paper establishes a coherent taxonomy based on the interaction mechanisms between gradient-based optimization and population-based selection.

To understand the diverse landscape of hybrid algorithms, the paper proposes a generic framework that captures the essential loops of both DRL and EA. In this framework, a population of agents interacts with an environment to generate trajectories. These trajectories serve two purposes: they provide the fitness signal for evolutionary selection and variation, and they populate a data collection mechanism (like a replay buffer) for RL-based learning.

A generic framework for combining Evolutionary Algorithms and Reinforcement Learning.

Categorizing Hybrid Approaches

The survey organizes the literature into four primary categories based on the target of the evolutionary process:

1. Evolution of Policies for Performance

These methods run evolutionary and DRL loops in parallel. A prominent technique is Actor Injection, as seen in Evolutionary Reinforcement Learning (ERL), where a DRL-trained agent is periodically inserted into the evolutionary population to accelerate progress. Other methods, such as CEM-RL, apply DRL gradient steps directly to population members to refine their parameters.

Architectures for (a) Evolutionary Reinforcement Learning (ERL) and (b) Proximal Distilled Evolutionary Reinforcement Learning (PDERL).

2. Evolution of Actions

Instead of evolving entire policies, some algorithms use evolutionary strategies (e.g., Cross-Entropy Method or PSO) to determine the optimal action for a given state. This is particularly effective in continuous action spaces where maximizing a Q-function is computationally demanding.

3. Evolution for Diversity

Methods focusing on Novelty Search and Quality-Diversity (QD) maintain populations of behaviorally diverse agents. This prevents convergence to local optima and is essential for solving tasks with sparse or deceptive rewards.

4. Evolution of Hyperparameters and Structure

Evolution is also used to optimize components inaccessible to gradients, including neural network architectures, reward functions, robot morphology, and hyperparameters (as in Population Based Training).

Key Insights and Discussion

  • Evolution as a Safety Filter: Evolutionary selection acts as a natural filter; while DRL might suggest a detrimental gradient update, the selection process discards failed individuals, ensuring the population only moves toward improved performance.
  • The Critic as a Bridge: In many hybrids, the DRL "Critic" serves as a surrogate fitness function, allowing the evolutionary loop to evaluate "virtual" individuals without expensive environment simulations.
  • Complementary Strengths: Evolution excels at exploring discrete structures and global landscapes, while DRL is superior at fine-tuning continuous weights within those structures.

Conclusion and Future Directions

While hybrid methods like CERL and PDERL show superior sample efficiency and stability over pure EA or DRL, the field lacks a unified benchmark for rigorous comparison. Future work should focus on "combining the combinations"—integrating surrogate fitness, actor injection, and diversity seeking into single architectures—and moving toward fully asynchronous, population-scale learning to leverage modern computing clusters.

Summary updated

Abstract

Deep neuroevolution and deep Reinforcement Learning have received a lot of attention in the last years. Some works have compared them, highlighting theirs pros and cons, but an emerging trend consists in combining them so as to benefit from the best of both worlds. In this paper, we provide a survey of this emerging trend by organizing the literature into related groups of works and casting all the existing combinations in each group into a generic framework. We systematically cover all easily available papers irrespective of their publication status, focusing on the combination mechanisms rather than on the experimental results. In total, we cover 45 algorithms more recent than 2017. We hope this effort will favor the growth of the domain by facilitating the understanding of the relationships between the methods, leading to deeper analyses, outlining missing useful comparisons and suggesting new combinations of mechanisms.
BibTeX
@article{sigaud2023combining,
  author = {Olivier Sigaud},
  title = {{Combining Evolution and Deep Reinforcement Learning for Policy Search: a Survey}},
  journal = {ACM Trans. Evol. Learn. Optim.},
  year = {2023},
  doi = {10.1145/3569096},
}
Figure