Combating catastrophic forgetting with developmental compression

Shawn L. E. Beaulieu, Sam Kriegman, Josh C. Bongard

Abstract

Catastrophic forgetting is a notorious problem in machine learning where training a model on new tasks destroys its performance on old ones. This paper introduces "developmental compression," an evolutionary approach that prevents forgetting by allowing an agent's neural network to "develop" (change weights) over its lifetime, gradually compressing specialized, task-specific networks into a single, robust generalist network without increasing the model's size.

The Challenge of Catastrophic Forgetting

If you train a neural network to play Pong, and then train that exact same network to play Tetris, it will likely forget how to play Pong. This phenomenon, known as catastrophic forgetting (or catastrophic interference), is a massive hurdle for creating generally intelligent agents that can operate across multiple environments.

Historically, researchers have tackled this by using modular networks or by freezing important weights. However, these approaches scale poorly: as the number of tasks grows, the network architecture must grow with it. The authors of this paper seek a more scalable solution that doesn't require engineering bespoke, ever-expanding architectures or relying on explicit domain knowledge to protect previously learned skills.

Experimental Setup

The authors test their approach using Evolutionary Robotics. They simulate a generic four-legged robot controlled by a simple neural network where sensors are directly connected to motors, with no hidden layers. The network's weights are optimized using a genetic algorithm.

The simulated four-legged robot in the Pyrosim environment.

The primary task is phototaxis, where the robot must walk toward a light-emitting box. The robot is sequentially placed in 2, 3, or 4 different environments, defined by the light's position (e.g., 30 body lengths in front of or behind the robot).

Developmental Compression (DC)

To combat forgetting, the authors introduce Developmental Compression (DC). Instead of a single weight matrix, an agent's genome contains a "base" controller and several "target" specialist controllers (one for each task environment).

  • Development: During a single evaluation (the robot's "lifetime"), the robot's brain develops. It starts the simulation using the base controller. As time ticks forward, the weights linearly interpolate toward the target specialist controller for that specific environment.
  • Compression: Over evolutionary time, mutations gently adjust these matrices. Because the robot is evaluated on how well it performs across all tasks, evolution gradually pulls the target matrices closer to the base matrix.
  • Result: Eventually, the specialized knowledge from the target matrices is "compressed" into the base controller, creating a single generalist network.

Key Results

DC was compared against a standard non-developmental evolutionary algorithm (Control) and a Random Search baseline.

Fitness metrics across 2, 3, and 4 environments. The Experiment (DC) treatment consistently maintains higher minimum fitness across all tasks compared to the Control, which tends to over-specialize.
  • DC successfully prevents catastrophic forgetting: In the 2-environment setup, DC significantly outperformed both the standard control and random search ($P < 0.0001$).
  • Better worst-case performance: The authors measured the "minimum fitness" (how well the robot did in its worst environment). The median minimum fitness for DC was significantly higher than for the control (e.g., 0.1103 vs 0.0781 in the 2-env setup).
  • Avoiding over-specialization: Standard control agents achieved high maximum fitness by over-specializing in one direction but completely failing when the light was placed behind them. DC agents, by contrast, steadily improved across all environments simultaneously.

Insights and Limitations

The success of DC relies on evolutionary canalization, where evolution uses target matrices as temporary "scaffolding" to learn a task before absorbing that knowledge into the base matrix. Additionally, the delayed onset of mutations in DC reduces the severity of mutations, protecting previously evolved capabilities.

However, scalability remains a limitation. While DC worked well for 2 environments, its ability to compress knowledge degraded when scaled to 3 and 4 environments. The authors attribute this to the simplicity of their genetic algorithm and the sparse neural controller. Future work will explore non-linear developmental trajectories and scaling to standard deep reinforcement learning benchmarks.

Summary updated

Abstract

Generally intelligent agents exhibit successful behavior across problems in several settings. Endemic in approaches to realize such intelligence in machines is catastrophic forgetting: sequential learning corrupts knowledge obtained earlier in the sequence, or tasks antagonistically compete for system resources. Methods for obviating catastrophic forgetting have sought to identify and preserve features of the system necessary to solve one problem when learning to solve another, or to enforce modularity such that minimally overlapping sub-functions contain task specific knowledge. While successful, both approaches scale poorly because they require larger architectures as the number of training instances grows, causing different parts of the system to specialize for separate subsets of the data. Here we present a method for addressing catastrophic forgetting called developmental compression. It exploits the mild impacts of developmental mutations to lessen adverse changes to previously-evolved capabilities and 'compresses' specialized neural networks into a generalized one. In the absence of domain knowledge, developmental compression produces systems that avoid overt specialization, alleviating the need to engineer a bespoke system for every task permutation and suggesting better scalability than existing approaches. We validate this method on a robot control problem and hope to extend this approach to other machine learning domains in the future.
BibTeX
@inproceedings{beaulieu2018combating,
  author = {Shawn L. E. Beaulieu and Sam Kriegman and Josh C. Bongard},
  title = {{Combating catastrophic forgetting with developmental compression}},
  booktitle = {GECCO},
  year = {2018},
  url = {https://openreview.net/forum?id=jYA8hh22wS},
}
Figure