Mitigating Catastrophic Forgetting with Physics-Indexed Specialist Memory: the Evolutionary Neuroplastic Agent

Blas Eugenio Vicco Zarate, Miguel Hernán ViccoarXiv 2026 · arXiv

TL;DR

The Evolutionary Neuroplastic Agent (ENA) is a gradient-free reinforcement learning architecture that combats catastrophic forgetting by archiving evolved "specialist" policies into a memory indexed by detected physical dynamics. By scaling search effort based on performance failure and using Bayesian policy reuse for retrieval, ENA maintains near-perfect retention across shifting environments without requiring explicit task labels.

What Problem Does This Paper Address?

Most reinforcement learning (RL) agents suffer from catastrophic forgetting: when an environment's dynamics change (e.g., a robot moves from a high-friction surface to a slippery one), the agent adapts by overwriting its existing neural network weights. When the original conditions return, the agent has "forgotten" its previous expertise and must learn from scratch.

While many continual learning methods exist, they often rely on explicit task boundaries (being told when the environment has changed) or complex regularization schemes. This paper investigates whether a gradient-free approach—using neuroevolution to build a "specialist memory"—can autonomously detect physical shifts and preserve a library of reusable skills.

Methods

The authors introduce the Evolutionary Neuroplastic Agent (ENA), which combines several modular skills:

  • Neuroevolutionary Search: Instead of gradient descent, ENA evolves a population of small neural network "specialists" through mutation and crossover.
  • Hall of Fame (Archive): Successful specialists are frozen and stored in an archive. Crucially, they are indexed by the "niche" (the specific physical regime) in which they were validated.
  • Environment Detection: ENA identifies changes in physics purely from experience.
  • In CartPole, it uses linear-Gaussian transition models to predict state changes.
  • In HalfCheetah, it uses a "chirp"—a fixed, diagnostic 64-step action sequence—to analyze the robot's mechanical response to its environment.
  • Failure-Driven Plasticity: ENA dynamically scales its search effort. If the active specialist is performing well, search effort is minimal. If performance drops (indicating a likely environment shift), "plasticity" increases, triggering broader and deeper evolutionary search.
  • Bayesian Policy Reuse (BPR): At test time, ENA uses a Bayesian framework to select which archived specialist to deploy, combining evidence from the environment detector with observed episodic rewards.

The overall training loop is summarized in the paper's algorithm: ENA Training Episode Algorithm

ENA Training Episode Algorithm

Datasets

The architecture was evaluated on two non-stationary benchmarks built on the Gymnasium framework:

  1. Non-stationary CartPole-v1: Five environments where gravity, pole mass, and pole length vary (e.g., "Mars-Light" vs. "Jupiter-Heavy").
  2. Non-stationary HalfCheetah-v5: Five environments with varying gravity, ground friction, and wind conditions.

In both cases, the agent trains on a sequence of three environments without being told when switches occur and is then tested on all five (including two unseen during training). The specific parameters for the CartPole environments are shown below: CartPole Environment Parameters

CartPole Environment Parameters

Key Insights and Ideas

  • Physics-based Indexing is Vital: Simply saving successful policies isn't enough. The "aha moment" here is that for effective recall, policies must be explicitly indexed by the physical regime (the "niche") they were learned in. Without this indexing, the agent struggles to distinguish which specialist fits the current task.
  • Evolution as a Natural Memory Source: Evolutionary methods naturally produce many candidate solutions during search. ENA turns this "by-product" into a structured memory, rendering policy archiving nearly free of dedicated training overhead.
  • The "Chirp" for System ID: For complex, contact-rich dynamics like HalfCheetah, standard transition models often fail. Using a fixed diagnostic perturbation (the chirp) proved to be a robust, gradient-free way to identify environmental physics.
  • Plasticity as Resource Management: By concentrating search effort only during performance failures, ENA achieves competitive retention while significantly reducing the total computational cost compared to constant full-effort search.

Key Results

  • Superior Retention: In CartPole, ENA-Combined achieved a mean success rate of 83.2% across test environments. Most importantly, it maintained near 100% retention of previously learned tasks. In contrast, standard learners like PPO and SAC suffered massive performance drops (e.g., PPO retained only ~4.3% on one task) when returning to earlier environments.
  • Autonomous Detection: ENA's detectors successfully identified environment shifts and recognized previously encountered regimes without any external supervision or labels.
  • Performance Trade-offs: While ENA excels at memory, it adapts more slowly than PPO on simple tasks and reaches lower peak returns than SAC on complex ones. This highlights a classic stability-plasticity trade-off.

The paper positions ENA against other state-of-the-art continual learning methods in the following table: Comparison of Related Methods

Comparison of Related Methods

Limitations and Future Work

  • Sample Efficiency: Like most evolutionary methods, ENA requires more environment interactions to reach peak performance compared to gradient-based methods.
  • Scaling to Large Networks: The current specialists are small (e.g., 32x16 hidden layers). Scaling neuroevolution to the massive networks used in modern Deep RL remains a challenge.
  • Environment Resets: The "chirp" and candidate evaluation mechanisms assume the environment can be reset to a known state, which may not be feasible in all real-world robotics applications.
  • Future Direction: The authors suggest a hybrid approach: using gradient-based learners (like SAC or PPO) as local optimizers within each niche, while using ENA’s physics-indexed "Hall of Fame" and failure-driven plasticity to manage long-term memory and adaptation.

Summary updated

Abstract

Continual reinforcement learning requires adaptation to changing conditions while preserving performance under previously encountered dynamics. It was investigated whether specialists generated by evolutionary search can provide a reusable memory when combined with environment detection and failure-driven search. The Evolutionary Neuroplastic Agent (ENA) evolves small neural networks without gradients, archives selected policies by detected dynamic regime, and retrieves them through Bayesian policy reuse. The architecture was evaluated in non-stationary CartPole and HalfCheetah environments. In the 30-replicate CartPole ablation, the complete architecture achieved a mean success rate of 83.2% across the five test environments. Among variants that learned the harder CartPole environments, the complete architecture showed the most consistent retention. A selection ablation also showed that the retrieval rule affected performance. In the evaluated sequences, ENA adapted more slowly than PPO on CartPole and achieved lower returns than SAC on HalfCheetah. The evidence concerns the complete architecture and does not isolate gradient-free optimization as the cause of retention or establish superiority over other continual-learning methods.
BibTeX
@misc{zarate2026mitigating,
  author = {Blas Eugenio Vicco Zarate and Miguel Hernán Vicco},
  title = {{Mitigating Catastrophic Forgetting with Physics-Indexed Specialist Memory: the Evolutionary Neuroplastic Agent}},
  year = {2026},
  eprint = {2611.21598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21598},
}
Figure