Large scale evolution of convolutional neural networks using volunteer computing

Travis Desell

TL;DR

This paper introduces EXACT (Evolutionary Exploration of Augmenting Convolutional Topologies), an algorithm that evolves the architecture of Convolutional Neural Networks (CNNs) from scratch. By leveraging the Citizen Science Grid (a BOINC-based volunteer computing project), the researchers harnessed over 4,500 home computers to train 120,000+ models, evolving "organic" network structures that achieved 98.32% accuracy on the MNIST dataset.

What Problem Does This Paper Address?

While CNNs have revolutionized computer vision, their architectures (the arrangement of layers, filters, and connections) are typically designed by human experts through trial and error. Automated design, or Neuroevolution, offers a way to discover better architectures, but it is computationally "expensive."

Standard neuroevolution techniques like NEAT work well for small, simple networks but struggle to scale to the massive parameter counts of CNNs. Furthermore, training a single CNN can take hours or days; training the thousands of candidates required for an evolutionary search requires a level of compute power usually reserved for tech giants. This paper seeks to democratize that process by scaling neuroevolution to distributed, volunteer-run computing environments.

Methods

The core contribution is the EXACT algorithm. It is inspired by the NEAT (Neuroevolution of Augmenting Topologies) algorithm but modified for the specific constraints of convolutional layers:

  • Topology Evolution: Instead of evolving individual neurons, EXACT evolves the connections between convolutional filters. It uses a mathematical observation that any two filters can be connected by a convolution if the filter size is calculated based on the difference between input and output dimensions.
  • Asynchronous Evolution: To handle thousands of volunteer computers with varying speeds and connection stabilities, the algorithm uses an asynchronous master-worker model. Workers request a "genome," train it, and report the fitness (accuracy) back to the master.
  • Mutation and Crossover: The algorithm uses specific operators to "Disable/Enable Edges," "Split Edges" (adding a new node in between), and "Change Node Size" (modifying filter dimensions).
  • Epigenetic Weight Initialization: A novel feature where child networks can optionally inherit the trained weights of their parent networks for unmutated parts of the topology, potentially giving the evolution a "head start."
  • Cross-Platform Validation: Because different CPUs and operating systems handle floating-point math differently, the authors implemented custom, deterministic versions of functions like exp() and random number generators to ensure that a model trained on Windows would produce the exact same results as one on Linux.

Datasets

The researchers used the MNIST handwritten digits dataset (60,000 training images, 10,000 testing images) as their primary benchmark. While MNIST is considered "solved" by modern standards, it served as a vital proof-of-concept for the high-computational demands of the EXACT algorithm.

Key Insights and Ideas

  • "Organic" Architectures: The most striking finding was that the evolved networks looked nothing like human-designed ones. While humans prefer neat, sequential layers (like LeNet or AlexNet), EXACT produced messy, highly interconnected structures with "vestigial" filters and edges that resemble the complex branching of biological neurons in the cerebral cortex.
  • The Epigenetic Advantage: Inheriting weights from parents (epigenetic initialization) led to better final accuracy compared to starting with random weights. Surprisingly, it didn't necessarily speed up the training process, but it helped the evolution find better-performing solutions.
  • Volunteer Power: The study proved that "Volunteer Computing" (BOINC) is a viable platform for deep learning research. By breaking the search into 120,000 independent training tasks, they achieved a scale of experimentation that would be prohibitively expensive on commercial cloud providers.

Key Results

  • Accuracy: The best-evolved networks reached 98.32% accuracy on MNIST.
  • Scale: Over a two-month period, 4,500 volunteered computers participated in the search.
  • Performance vs. Human Design: The evolved networks outperformed human-designed benchmarks (including a modified LeNet-5) when trained using the same rudimentary backpropagation strategy (ReLU units, L2 regularization, and Nesterov momentum).
  • Efficiency: The evolved networks often showed improved training and testing error rates compared to standard architectures with similar weight counts.

Limitations and Future Work

  • No Pooling: The current version of EXACT does not yet evolve pooling layers (max-pooling or average-pooling), which are standard in most modern CNNs.
  • 2D Constraints: The implementation was limited to 2D grayscale inputs (like MNIST). The authors acknowledge the need to update the algorithm to handle 3D color inputs for datasets like CIFAR-10 or ImageNet.
  • Fixed Hyperparameters: While the topology was evolved, the training hyperparameters (learning rate, momentum, etc.) were held constant. Future work could involve evolving these alongside the network structure.
  • Computational Intensity: Even with 4,500 computers, the search took two months for a relatively simple dataset, highlighting the massive overhead of architecture search.

Summary updated

Abstract

This work presents a new algorithm called evolutionary exploration of augmenting convolutional topologies (EXACT), which is capable of evolving the structure of convolutional neural networks (CNNs). EXACT is in part modeled after the neuroevolution of augmenting topologies (NEAT) algorithm, with notable exceptions to allow it to scale to large scale distributed computing environments and evolve networks with convolutional filters. In addition to multithreaded and MPI versions, EXACT has been implemented as part of a BOINC volunteer computing project, allowing large scale evolution. During a period of two months, over 4,500 volunteered computers on the Citizen Science Grid trained over 120,000 CNNs and evolved networks reaching 98.32% test data accuracy on the MNIST handwritten digits dataset. These results are even stronger as the backpropagation strategy used to train the CNNs was fairly rudimentary (ReLU units, L2 regularization and Nesterov momentum) and these were initial test runs done without refinement of the backpropagation hyperparameters. Further, the EXACT evolutionary strategy is independent of the method used to train the CNNs, so they could be further improved by advanced techniques like elastic distortions, pretraining and dropout. The evolved networks are also quite interesting, showing "organic" structures and significant differences from standard human designed architectures.
BibTeX
@inproceedings{desell2017large,
  author = {Travis Desell},
  title = {{Large scale evolution of convolutional neural networks using volunteer computing}},
  booktitle = {GECCO (Companion)},
  year = {2017},
  doi = {10.1145/3067695.3076002},
}

Loading related content...

Figure