TL;DR
This paper introduces Differentiable Pattern Producing Networks (DPPNs), a hybrid model combining evolutionary topology search with gradient-based weight learning. By using DPPNs to indirectly encode the weights of a neural network, the authors successfully compress a 157,000-parameter fully connected autoencoder into fewer than 200 parameters. Remarkably, the evolutionary process naturally rediscovers convolutional structures—like local receptive fields and weight sharing—without human design, leading to highly efficient and generalizable models.
What Problem Does This Paper Address?
In evolutionary computation, Compositional Pattern Producing Networks (CPPNs) are a popular way to encode complex, high-dimensional outputs (like images or neural network weights) using very few parameters. However, traditional CPPNs rely purely on evolution to optimize both their structure (topology) and their weights. While this works for small spaces, it scales poorly compared to the gradient-based methods (like backpropagation) used to train modern deep neural networks with millions of parameters.
Conversely, standard deep learning relies heavily on human-engineered architectures—like Convolutional Neural Networks (CNNs)—which embed strong priors about translation invariance. The authors ask: Can we combine the topology-searching power of evolution with the scalable weight-tuning of gradient descent to automatically discover optimal, highly compressed network architectures from the data itself?
Methods
The authors introduce the DPPN (Differentiable Pattern Producing Network). Unlike standard CPPNs, DPPNs are built with differentiable activation functions, allowing their weights to be trained via backpropagation and gradient descent (specifically, the Adam optimizer).
The optimization process uses a Lamarckian evolutionary algorithm:
- Evolution (Outer Loop): A population of DPPNs undergoes mutation (adding/removing nodes and edges) and crossover (merging hidden units from two parents).
- Learning (Inner Loop): During fitness evaluation, the DPPN's weights are trained using gradient descent on a specific task.
- Lamarckian Inheritance: Crucially, the learned weights are inherited by the offspring. The authors compared this to Darwinian (learned weights discarded) and Baldwinian approaches, finding Lamarckian evolution to be significantly more effective.
The DPPN is used as an indirect encoding for a larger network. Instead of learning the larger network's weights directly, the DPPN takes the spatial coordinates of two neurons in the larger network as input and outputs the connection weight between them.
Datasets
- MNIST: A standard dataset of handwritten digits. Used to train a denoising autoencoder (reconstructing digits where 10% of pixels were dropped).
- Omniglot: A dataset of handwritten characters from various alphabets. Used strictly for testing to evaluate how well the representations learned on MNIST generalize to unseen character types.
Key Insights and Ideas
- Evolution Discovers Convolution: The most striking finding is that when tasked with compressing a fully connected network, the DPPN naturally evolves weight-sharing and local receptive fields. It essentially "invents" the convolutional architecture from scratch. The learned weights visually resemble the regular on-center and off-center receptive fields found in biological retinal ganglion cells.
- Extreme Regularization: Because the DPPN acts as a massive bottleneck—forcing a massive weight matrix to be generated by a tiny mathematical graph—it acts as a powerful regularizer. It forces the network to learn smooth, generalizable patterns rather than memorizing noise.
- Lamarckian Evolution Wins: Inheriting acquired characteristics (the weights learned via gradient descent) drastically speeds up the evolutionary process and results in better final performance compared to standard Darwinian evolution.
- Crossover is Crucial: Recombining the hidden units of two parent DPPNs yielded an order-of-magnitude improvement in Mean Squared Error (MSE) compared to mutation alone, likely because it allows the merging of useful sub-features discovered by different individuals.
Key Results
- Massive Compression: The DPPN successfully compressed the parameters of a fully connected denoising autoencoder from 157,684 parameters down to just 187 parameters.
- High Performance: Despite the extreme compression, the 187-parameter DPPN achieved a Binary Cross Entropy (BCE) of 0.09 on the MNIST test set.
- Superior Generalization: When tested on the Omniglot dataset (after only training on MNIST), the DPPN-encoded network achieved a BCE of 0.121. This outperformed an equivalent directly-encoded network, proving that the evolutionary bottleneck forces the learning of highly generalizable features.
Limitations and Future Work
While the results are a fascinating proof-of-concept, the experiments are limited to relatively simple tasks (MNIST autoencoding) and shallow networks compared to state-of-the-art deep learning models.
For future work, the authors suggest scaling this framework to train much deeper neural networks. They also note that the symbiosis of evolution and gradient-based learning holds significant potential for helping optimization algorithms avoid getting stuck in local optima or saddle points, a common issue in standard deep learning.
Summary updated