Abstract
In addition to their undisputed success in solving classical optimization problems, neuroevolutionary and population-based algorithms have become an alternative to standard reinforcement learning methods. However, evolutionary methods often lack the sample efficiency of standard value-based methods that leverage gathered state and value experience. If reinforcement learning for real-world problems with significant resource cost is considered, sample efficiency is essential. The enhancement of evolutionary algorithms with experience exploiting methods is thus desired and promises valuable insights. This work presents a hybrid algorithm that combines topology-changing neuroevolutionary optimization with value-based reinforcement learning. We illustrate how the behavior of policies can be used to create distance and loss functions, which benefit from stored experiences and calculated state values. They allow us to model behavior and perform a directed search in the behavior space by gradient-free evolutionary algorithms and surrogate-based optimization. For this purpose, we consolidate different methods to generate and optimize agent policies, creating a diverse population. We exemplify the performance of our algorithm on standard benchmarks and a purpose-built real-world problem. Our results indicate that combining methods can enhance the sample efficiency and learning speed for evolutionary approaches.
Introduction
Neuroevolution (NE) offers a powerful framework for optimizing both the weights and topologies of neural networks. However, traditional NE methods often suffer from low sample efficiency, as they typically treat entire episodes as single data points. In contrast, value-based reinforcement learning (RL) methods like PPO or A2C leverage step-by-step experience but are generally restricted to fixed network architectures. This paper introduces BNET (Behavior-based Neuroevolutionary Training), a hybrid framework that combines the structural flexibility of NE with the data-driven efficiency of value-based RL.
Methodology: The BNET Framework
BNET utilizes Cartesian Genetic Programming (CGP) to represent neural networks as arbitrary, non-layered graphs. The core innovation lies in its multi-path search strategy, which populates the next generation of agents through three distinct mechanisms:
- Fitness-based Search: A standard evolutionary path that selects "champions" based on cumulative reward. It incorporates a robust selection mechanism to mitigate noise in stochastic environments.
- Behavior-based Search: This path utilizes an archive of elite experiences and a value-critic model to calculate advantage-weighted behavior distances. By minimizing these distances, the algorithm performs a directed search in behavior space, effectively mimicking successful actions from previous trials.
- Surrogate-based Search: BNET employs Surrogate Model-Based Optimization (SMBO) using a Kriging model. Unlike traditional approaches that use genotypic kernels, BNET uses a behavioral distance kernel, predicting performance based on how an agent interacts with the environment.
Experimental Setup
The framework was evaluated across three benchmarks:
- CartPole-v0: A standard control task.
- MountainCar-v0: A sparse-reward environment that is notoriously difficult for gradient-based methods.
- Robot Maze: A custom real-world benchmark involving a UR10e robotic arm tilting a physical Lego maze to guide a marble to a goal.
Results and Discussion
BNET demonstrated significant improvements in sample efficiency across all tasks. In CartPole-v0, the full BNET framework solved the task in a median of 3,576 steps, compared to approximately 17,000 steps for pure mutation-based NE. In the challenging MountainCar-v0 environment, BNET consistently found solutions within 13.5k steps, while standard PPO and A2C baselines failed to solve it within the same budget.
The real-world Robot Maze results further validated the approach. BNET variants combining behavior distance and cross-entropy outperformed PPO, reaching the target in fewer timesteps (Mean: 1156 vs 1580). These results suggest that focusing on behavioral embeddings rather than genotypic structures allows for more accurate surrogate modeling and more effective directed search in topology-changing networks.
Conclusion
BNET successfully bridges the gap between neuroevolution and value-based reinforcement learning. By integrating behavior-based search and surrogate modeling, the framework achieves high sample efficiency while maintaining the ability to optimize network topologies. Future work will focus on extending the framework to continuous action spaces and exploring the potential for "jumpstarting" the population with offline demonstration data.
Summary updated