Convolution by Evolution: Differentiable Pattern Producing Networks

Chrisantha Fernando, Dylan Banarse, Malcolm Reynolds, Frederic Besse, David Pfau, Max Jaderberg, Marc Lanctot, Daan Wierstra

Abstract

In this work we introduce a differentiable version of the Compositional Pattern Producing Network, called the DPPN. Unlike a standard CPPN, the topology of a DPPN is evolved but the weights are learned. A Lamarckian algorithm, that combines evolution and learning, produces DPPNs to reconstruct an image. Our main result is that DPPNs can be evolved/trained to compress the weights of a denoising autoencoder from 157684 to roughly 200 parameters, while achieving a reconstruction accuracy comparable to a fully connected network with more than two orders of magnitude more parameters. The regularization ability of the DPPN allows it to rediscover (approximate) convolutional network architectures embedded within a fully connected architecture. Such convolutional architectures are the current state of the art for many computer vision applications, so it is satisfying that DPPNs are capable of discovering this structure rather than having to build it in by design. DPPNs exhibit better generalization when tested on the Omniglot dataset after being trained on MNIST, than directly encoded fully connected autoencoders. DPPNs are therefore a new framework for integrating learning and evolution.

Introduction

Modern deep learning relies heavily on human-engineered architectures, such as Convolutional Neural Networks (CNNs), which embed strong priors like translation invariance. In contrast, evolutionary computation often uses Compositional Pattern Producing Networks (CPPNs) to encode complex structures with few parameters. However, traditional CPPNs scale poorly because they rely on evolution to optimize both topology and weights.

This paper introduces Differentiable Pattern Producing Networks (DPPNs), a hybrid model that combines evolutionary topology search with gradient-based weight learning. The goal is to determine if evolution can automatically discover optimal, highly compressed network architectures—like convolution—directly from data.

The DPPN Framework

The DPPN is a differentiable version of the CPPN. Unlike its predecessor, its activation functions are differentiable, allowing weights to be optimized via backpropagation using the Adam optimizer. The system utilizes a Lamarckian evolutionary algorithm where learned weights are inherited by offspring, significantly accelerating the optimization process.

The four-step Lamarckian training process for DPPNs.

The DPPN acts as an indirect encoding for a larger target network. Instead of learning the target network's weights directly, the DPPN takes spatial coordinates as input and outputs the connection weight between neurons. This creates a massive bottleneck that forces the model to learn smooth, generalizable patterns.

Results: Evolution Discovers Convolution

The most significant finding is that when compressing a fully connected denoising autoencoder, the DPPN naturally evolves weight-sharing and local receptive fields. Without any human-designed convolutional priors, the evolutionary process "invents" convolutional structures from scratch.

Evolved weight patterns resembling biological on-center and off-center receptive fields.

As shown in Figure 2, the generated weights visually resemble on-center and off-center receptive fields found in biological vision systems. This extreme regularization allowed the authors to compress a 157,684-parameter network into just 187 parameters while maintaining high performance on MNIST (Binary Cross Entropy of 0.09).

Generalization and Performance

The DPPN's structural bottleneck serves as a powerful regularizer. When trained on MNIST and tested on the unseen Omniglot dataset, the DPPN-encoded network achieved a BCE of 0.121, outperforming directly-encoded networks. This suggests that the evolved representations are inherently more generalizable than those learned through direct weight optimization.

Furthermore, the inclusion of crossover in the evolutionary loop provided an order-of-magnitude improvement in performance, as it allowed the model to recombine useful sub-features discovered by different individuals in the population.

Conclusion

DPPNs represent a promising step toward integrating evolution and gradient-based learning. By allowing evolution to handle topology and backpropagation to handle weight tuning, the model successfully rediscovers state-of-the-art architectural features like convolution. Future work will focus on scaling this approach to deeper networks and exploring its potential to navigate complex optimization landscapes.

Summary updated

Abstract

In this work we introduce a differentiable version of the Compositional Pattern Producing Network, called the DPPN. Unlike a standard CPPN, the topology of a DPPN is evolved but the weights are learned. A Lamarckian algorithm, that combines evolution and learning, produces DPPNs to reconstruct an image. Our main result is that DPPNs can be evolved/trained to compress the weights of a denoising autoencoder from 157684 to roughly 200 parameters, while achieving a reconstruction accuracy comparable to a fully connected network with more than two orders of magnitude more parameters. The regularization ability of the DPPN allows it to rediscover (approximate) convolutional network architectures embedded within a fully connected architecture. Such convolutional architectures are the current state of the art for many computer vision applications, so it is satisfying that DPPNs are capable of discovering this structure rather than having to build it in by design. DPPNs exhibit better generalization when tested on the Omniglot dataset after being trained on MNIST, than directly encoded fully connected autoencoders. DPPNs are therefore a new framework for integrating learning and evolution.
BibTeX
@inproceedings{fernando2016convolution,
  author = {Chrisantha Fernando and Dylan Banarse and Malcolm Reynolds and Frederic Besse and David Pfau and Max Jaderberg and Marc Lanctot and Daan Wierstra},
  title = {{Convolution by Evolution: Differentiable Pattern Producing Networks}},
  booktitle = {GECCO},
  year = {2016},
  url = {https://openreview.net/forum?id=6qX6_nnvuVm},
}
Figure