Abstract
Catastrophic forgetting is a notorious problem in machine learning where training a model on new tasks destroys its performance on old ones. This paper introduces "developmental compression," an evolutionary approach that prevents forgetting by allowing an agent's neural network to "develop" (change weights) over its lifetime, gradually compressing specialized, task-specific networks into a single, robust generalist network without increasing the model's size.
The Challenge of Catastrophic Forgetting
If you train a neural network to play Pong, and then train that exact same network to play Tetris, it will likely forget how to play Pong. This phenomenon, known as catastrophic forgetting (or catastrophic interference), is a massive hurdle for creating generally intelligent agents that can operate across multiple environments.
Historically, researchers have tackled this by using modular networks or by freezing important weights. However, these approaches scale poorly: as the number of tasks grows, the network architecture must grow with it. The authors of this paper seek a more scalable solution that doesn't require engineering bespoke, ever-expanding architectures or relying on explicit domain knowledge to protect previously learned skills.
Experimental Setup
The authors test their approach using Evolutionary Robotics. They simulate a generic four-legged robot controlled by a simple neural network where sensors are directly connected to motors, with no hidden layers. The network's weights are optimized using a genetic algorithm.
The primary task is phototaxis, where the robot must walk toward a light-emitting box. The robot is sequentially placed in 2, 3, or 4 different environments, defined by the light's position (e.g., 30 body lengths in front of or behind the robot).
Developmental Compression (DC)
To combat forgetting, the authors introduce Developmental Compression (DC). Instead of a single weight matrix, an agent's genome contains a "base" controller and several "target" specialist controllers (one for each task environment).
- Development: During a single evaluation (the robot's "lifetime"), the robot's brain develops. It starts the simulation using the base controller. As time ticks forward, the weights linearly interpolate toward the target specialist controller for that specific environment.
- Compression: Over evolutionary time, mutations gently adjust these matrices. Because the robot is evaluated on how well it performs across all tasks, evolution gradually pulls the target matrices closer to the base matrix.
- Result: Eventually, the specialized knowledge from the target matrices is "compressed" into the base controller, creating a single generalist network.
Key Results
DC was compared against a standard non-developmental evolutionary algorithm (Control) and a Random Search baseline.
- DC successfully prevents catastrophic forgetting: In the 2-environment setup, DC significantly outperformed both the standard control and random search ($P < 0.0001$).
- Better worst-case performance: The authors measured the "minimum fitness" (how well the robot did in its worst environment). The median minimum fitness for DC was significantly higher than for the control (e.g., 0.1103 vs 0.0781 in the 2-env setup).
- Avoiding over-specialization: Standard control agents achieved high maximum fitness by over-specializing in one direction but completely failing when the light was placed behind them. DC agents, by contrast, steadily improved across all environments simultaneously.
Insights and Limitations
The success of DC relies on evolutionary canalization, where evolution uses target matrices as temporary "scaffolding" to learn a task before absorbing that knowledge into the base matrix. Additionally, the delayed onset of mutations in DC reduces the severity of mutations, protecting previously evolved capabilities.
However, scalability remains a limitation. While DC worked well for 2 environments, its ability to compress knowledge degraded when scaled to 3 and 4 environments. The authors attribute this to the simplicity of their genetic algorithm and the sparse neural controller. Future work will explore non-linear developmental trajectories and scaling to standard deep reinforcement learning benchmarks.
Summary updated