When Does Neuroevolution Outcompete Reinforcement Learning in Transfer Learning Tasks?

Eleni Nisioti, Erwan Plantec, Milton Montero, Joachim Winther Pedersen, Sebastian Risi

Introduction

Reinforcement Learning (RL) has become the standard for high-dimensional control, yet it remains notoriously brittle in transfer learning scenarios. When an agent must transition from simple to complex tasks—a process known as a curriculum—RL often suffers from non-stationarity. Shifting goalposts can lead to "catastrophic forgetting," where gradient-based updates for new tasks erase previously mastered skills. This paper investigates whether Neuroevolution (NE), which optimizes through population-based search rather than gradients, offers a more robust alternative for sequential task learning.

Benchmarking Transfer Learning

To evaluate these approaches, the authors developed two JAX-based benchmarks specifically designed for transfer learning: Stepping Gates and Ecorobot.

Overview of the Ecorobot benchmark, featuring various robot morphologies and task environments like the Stepping Stones Maze and Hierarchical Obstacles.

Stepping Gates focuses on digital circuit emulation, requiring agents to function as logic gates (e.g., XOR, NAND). This includes N-parity tasks to test modular repetition and a Simple ALU task requiring modified repetitions. Ecorobot extends the Brax physics engine, challenging agents with tasks like the Stepping Stones Maze and locomotion over obstacles of increasing height.

The Stepping Gates benchmark, illustrating the N-parity and Simple ALU logic tasks used to test incremental skill acquisition.

Methodological Comparison

The study benchmarks several Neuroevolution variants against Proximal Policy Optimization (PPO), a state-of-the-art RL algorithm:

  • NEAT (NeuroEvolution of Augmenting Topologies): Evolves both network architecture and weights, starting small and adding complexity.
  • HyperNEAT: Uses indirect encoding (CPPNs) to generate large-scale weight patterns.
  • CMA-ES: A weight-optimization strategy for fixed architectures.
  • MAP-Elites: A Quality-Diversity algorithm focused on behavioral niches.

Key Insights: The Curriculum Paradox

A central finding is the "Curriculum Paradox" observed in RL. In the N-parity task, PPO failed to progress through the curriculum. However, an ablation study revealed that if PPO was presented with the final, most difficult task immediately, it could solve it. This indicates that the failure was not due to task difficulty, but rather the process of transfer, which caused the agent to unlearn fundamentals while attempting to adapt to new stages.

In contrast, Architecture Evolution proved to be a significant advantage. NEAT’s ability to "grow" its network protected earlier knowledge by adding new capacity for new challenges. While HyperNEAT excelled at escaping local optima (solving deceptive mazes that trapped PPO 100% of the time), it struggled with incremental fine-tuning, suggesting a trade-off between global search and precise skill transfer.

Results and Performance

  • Logic and Navigation: NEAT and CMA-ES solved all levels of the N-parity benchmark, whereas PPO’s performance plummeted after initial levels. In the Stepping Stones Maze, NEAT reached five out of seven stones on average, while PPO consistently failed at the first stone.
  • Locomotion: In the HalfCheetah obstacle course, NEAT evolved stable policies for jumping or flipping over walls. PPO typically produced unstable "bumpy" policies that led to failure as obstacle height increased.

Conclusion

Neuroevolution, particularly methods that evolve topology like NEAT, demonstrates superior resilience in transfer learning compared to standard RL. By maintaining a population of solutions and incrementally growing network structures, NE avoids the catastrophic forgetting inherent in gradient-based curriculum learning. Future research aims to combine the pattern-recognition strengths of indirect encodings with the structural growth of direct encodings to scale these benefits to complex robotic morphologies.

Summary updated

BibTeX
@inproceedings{nisioti2025when,
  author = {Eleni Nisioti and Erwan Plantec and Milton Montero and Joachim Winther Pedersen and Sebastian Risi},
  title = {{When Does Neuroevolution Outcompete Reinforcement Learning in Transfer Learning Tasks?}},
  booktitle = {GECCO},
  year = {2025},
  doi = {10.1145/3712256.3726475},
}
Figure