ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation

Pengyi Li, Hongyao Tang, Jianye Hao, Yan Zheng, Xian Fu, Zhaopeng Meng

TL;DR

ERL-Re$^2$ is a novel hybrid framework that combines the exploration power of Evolutionary Algorithms (EA) with the sample efficiency of Reinforcement Learning (RL) by using a shared nonlinear state representation and individual linear policy weights. This "two-scale" approach enables stable, semantically meaningful behavior-level evolution and collective knowledge sharing, setting a new state-of-the-art in complex continuous control tasks.

What Problem Does This Paper Address?

Integrating Deep RL and EA is a promising frontier: RL provides efficient gradient-based updates, while EA offers robust exploration and stability. However, existing hybrid methods often hit a performance ceiling. The authors identify two critical bottlenecks:

  1. Redundant Learning: In current frameworks, every agent (both RL and the EA population) learns its own independent policy network. This means they all have to redundantly learn the same environmental features, which is inefficient and slows down convergence.
  2. Unstable Evolution: Traditional EA operates by mutating or crossing over raw neural network parameters (weights and biases). Because deep networks are highly nonlinear, small changes in parameters can lead to catastrophic "policy crashes" where the agent's behavior completely breaks down.

Methods

The core innovation of ERL-Re$^2$ is its Two-scale Representation. Instead of full independent networks, all agents share a single nonlinear shared state representation ($Z_\phi$)—a deep feature extractor. On top of this, each agent maintains its own individual linear policy representation ($W$).

Figure 1: The left represents ERL framework and the right rep- resents ERL-Re2 framework. In ERL-Re2, all the policies are composed of the nonlinear shared state representation Zϕ and an individual linear policy representation W.

Key Components:

  • Collective Representation Learning: The shared feature extractor is updated using a "unifying gradient direction" derived from the value functions of all agents. This ensures the representation captures features that are useful for the entire population, not just a single agent.
  • Behavior-Level Genetic Operators: Because the policies are linear, each row of the weight matrix $W$ corresponds to a specific action dimension. This allows for novel behavior-level crossover (swapping weights for specific actions between parents) and mutation (perturbing specific action weights). These operators have clear semantic meaning and are far more stable than parameter-level changes.
  • Efficient Fitness Estimation: To avoid the high cost of full environment rollouts, the authors use a Policy-extended Value Function Approximator (PeVFA). This allows them to estimate an agent's fitness using short, $H$-step bootstrapped rollouts, significantly improving sample efficiency.
Figure 1: The left represents ERL framework and the right rep- resents ERL-Re2 framework. In ERL-Re2, all the policies are composed of the nonlinear shared state representation Zϕ and an individual linear policy representation W.

Datasets

The researchers evaluated ERL-Re$^2$ on several challenging benchmarks:

  • MuJoCo: Six standard continuous control tasks (HalfCheetah, Swimmer, Hopper, Ant, Walker, and Humanoid).
  • DeepMind Control Suite (DMC): Four tasks using pixel-level image inputs to test the framework's ability to handle high-dimensional visual states.
  • StarCraft II (SMAC): Six multi-agent micromanagement tasks to demonstrate scalability to complex, high-dimensional joint action spaces.

Key Insights and Ideas

  • Decoupling "What" from "How": By separating the complex task of understanding the environment (nonlinear representation) from the task of deciding an action (linear policy), the framework allows the population to collaborate on the former while competing on the latter.
  • Linearity is a Feature, Not a Bug: While deep networks are usually preferred for their power, keeping the final policy layer linear makes evolutionary operators predictable and semantically grounded. You can change how an agent "kicks" without accidentally breaking how it "balances."
  • Shared Value Maximization: Updating the shared representation to maximize the value functions of all agents ensures that as the population evolves, the underlying "understanding" of the world evolves to support those new behaviors.
Figure 3: Behavior-level crossover and mutation op- erators in ERL-Re2. The offspring and parents are indexed by c1, c2, m1, m2 and p1, p2 respectively.

Key Results

ERL-Re$^2$ consistently outperformed advanced baselines like SAC, PPO, and previous hybrid methods (ERL, PDERL, CEM-RL).

  • SOTA Performance: It achieved state-of-the-art results across all six MuJoCo tasks. Notably, it is the first algorithm to significantly outperform pure EA on the notoriously difficult "Swimmer" task.
  • Convergence Speed: In tasks like "Ant," ERL-Re$^2$ showed a 5x improvement in convergence rate compared to standard EA.
  • Parameter Efficiency: Because agents share the large feature extractor, ERL-Re$^2$ is much more scalable. For a population of 100 agents on HalfCheetah, it requires ~171k additional parameters, whereas traditional methods would require over 1 billion.
Figure 5: Performance comparison between ERL-Re2 and baselines (all in the DDPG version).

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Theoretical Grounding: The work is currently an empirical proof of concept; formal proofs for optimality and convergence are still needed.
  • Diversity: The current version doesn't have an explicit mechanism to maintain population diversity, which is a core strength of many EA methods. Integrating "Quality-Diversity" principles is a suggested next step.
  • Model-Based Extensions: The framework is currently model-free. Exploring how to integrate world models could further boost sample efficiency.

Summary updated

Abstract

Deep Reinforcement Learning (Deep RL) and Evolutionary Algorithms (EA) are two major paradigms of policy optimization with distinct learning principles, i.e., gradient-based v.s. gradient-free. An appealing research direction is integrating Deep RL and EA to devise new methods by fusing their complementary advantages. However, existing works on combining Deep RL and EA have two common drawbacks: 1) the RL agent and EA agents learn their policies individually, neglecting efficient sharing of useful common knowledge; 2) parameter-level policy optimization guarantees no semantic level of behavior evolution for the EA side. In this paper, we propose Evolutionary Reinforcement Learning with Two-scale State Representation and Policy Representation (ERL-Re$^2$), a novel solution to the aforementioned two drawbacks. The key idea of ERL-Re$^2$ is two-scale representation: all EA and RL policies share the same nonlinear state representation while maintaining individual} linear policy representations. The state representation conveys expressive common features of the environment learned by all the agents collectively; the linear policy representation provides a favorable space for efficient policy optimization, where novel behavior-level crossover and mutation operations can be performed. Moreover, the linear policy representation allows convenient generalization of policy fitness with the help of the Policy-extended Value Function Approximator (PeVFA), further improving the sample efficiency of fitness estimation. The experiments on a range of continuous control tasks show that ERL-Re$^2$ consistently outperforms advanced baselines and achieves the State Of The Art (SOTA). Our code is available on https://github.com/yeshenpy/ERL-Re2.
BibTeX
@inproceedings{li2023erl,
  author = {Pengyi Li and Hongyao Tang and Jianye Hao and Yan Zheng and Xian Fu and Zhaopeng Meng},
  title = {{ERL-Re2: Efficient Evolutionary Reinforcement Learning with Shared State Representation and Individual Policy Representation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=FYZCHEtt6H0},
}
Figure