Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing

Travis Desell

Abstract

This paper introduces Evolutionary Exploration of Augmenting Convolutional Topologies (EXACT), a novel neuroevolutionary algorithm designed to evolve the architecture of Convolutional Neural Networks (CNNs). By adapting the principles of NEAT for large-scale distributed environments, EXACT enables the discovery of high-performing, non-traditional network topologies. Utilizing the Citizen Science Grid and over 4,500 volunteer computers, the algorithm trained more than 120,000 CNNs, achieving a 98.32% accuracy on the MNIST dataset. The evolved architectures exhibit "organic" structures significantly different from human-designed models, suggesting new directions for automated neural architecture search.

Introduction

The success of deep learning, particularly Convolutional Neural Networks (CNNs), has largely depended on architectures designed by human experts through iterative trial and error. While effective, this manual design process is limited by human intuition and the immense search space of possible configurations. Neuroevolution—the use of evolutionary algorithms to design neural networks—offers a path toward automated architecture discovery. However, traditional methods like Neuroevolution of Augmenting Topologies (NEAT) struggle to scale to the complexity of CNNs and the computational demands of training them.

This research presents EXACT, an algorithm specifically engineered to evolve CNN topologies at scale. By leveraging volunteer computing via the Berkeley Open Infrastructure for Network Computing (BOINC), the study demonstrates that massive-scale neuroevolution is accessible beyond the confines of high-performance computing centers.

The EXACT Algorithm

EXACT evolves the connections between convolutional filters rather than individual neurons. It addresses the scaling challenges of NEAT through several key innovations:

Topology Evolution and Mutation

EXACT treats the network as a directed acyclic graph where nodes represent convolutional filters and edges represent connections. The algorithm employs specific mutation operators:

  • Edge Modification: Enabling or disabling connections between filters.
  • Node Splitting: Inserting a new filter node between existing connections.
  • Node Resizing: Adjusting the dimensions of convolutional filters.

A critical mathematical observation allows any two filters to be connected by a convolution, provided the filter size is calculated based on the difference between input and output dimensions.

Asynchronous Distributed Computing

To operate within a volunteer computing environment, EXACT utilizes an asynchronous master-worker model. Unlike synchronous evolution, which requires all members of a generation to finish before proceeding, EXACT allows workers to request genomes, perform training, and report fitness independently. This handles the inherent volatility of volunteer hardware, where nodes may disconnect or vary significantly in processing speed.

Epigenetic Weight Initialization

To improve evolutionary efficiency, EXACT implements epigenetic weight initialization. Child networks can optionally inherit trained weights from their parents for unmutated parts of the topology. This "head start" allows the evolutionary process to focus on optimizing new structures rather than retraining the entire network from scratch.

Experimental Setup and Volunteer Computing

The study utilized the Citizen Science Grid, a BOINC-based project. Over a two-month period, approximately 4,500 volunteers contributed their computing power. To ensure consistency across diverse hardware and operating systems, the researchers implemented deterministic mathematical functions (e.g., exp()) and random number generators, ensuring that a model's fitness would be identical regardless of the host machine.

The MNIST dataset (60,000 training and 10,000 testing images) served as the primary benchmark. While MNIST is a standard dataset, the sheer volume of models trained (over 120,000) provided a rigorous test of the EXACT framework's scalability.

Results and Discussion

The evolutionary process yielded several significant findings:

Performance

The best-evolved networks reached a test accuracy of 98.32%. This result is particularly notable because the training regime used only basic backpropagation techniques—ReLU units, L2 regularization, and Nesterov momentum—without advanced optimizations like dropout or elastic distortions. When compared to human-designed benchmarks like LeNet-5 under the same training constraints, the EXACT-evolved networks demonstrated superior performance and efficiency.

Organic Architectures

One of the most compelling outcomes was the visual structure of the evolved networks. Unlike the sequential, layered approach favored by human designers, EXACT produced "organic" topologies. These networks featured complex branching, highly interconnected nodes, and "vestigial" structures reminiscent of biological neural pathways in the cerebral cortex.

The Role of Epigenetics

Experiments confirmed that epigenetic weight inheritance led to higher final accuracy. While it did not significantly reduce the time required for individual training runs, it allowed the evolutionary search to reach higher fitness peaks by preserving learned features across generations.

Limitations and Future Directions

While EXACT proves the viability of volunteer-driven neuroevolution, several areas for improvement remain:

  • Pooling Layers: The current implementation does not evolve pooling layers, which are standard for dimensionality reduction in CNNs.
  • Input Dimensionality: The system was tested on 2D grayscale images. Expanding to 3D color inputs (e.g., CIFAR-10) is a necessary next step.
  • Hyperparameter Evolution: Future iterations could evolve training hyperparameters (learning rates, momentum) alongside the network topology.

Conclusion

EXACT demonstrates that evolutionary algorithms can successfully design complex CNN architectures when paired with large-scale distributed computing. By harnessing the power of thousands of volunteers, this approach democratizes neural architecture search and reveals "organic" design patterns that challenge traditional human-centric engineering.

Summary updated

Abstract

This work presents a new algorithm called evolutionary exploration of augmenting convolutional topologies (EXACT), which is capable of evolving the structure of convolutional neural networks (CNNs). EXACT is in part modeled after the neuroevolution of augmenting topologies (NEAT) algorithm, with notable exceptions to allow it to scale to large scale distributed computing environments and evolve networks with convolutional filters. In addition to multithreaded and MPI versions, EXACT has been implemented as part of a BOINC volunteer computing project, allowing large scale evolution. During a period of two months, over 4,500 volunteered computers on the Citizen Science Grid trained over 120,000 CNNs and evolved networks reaching 98.32% test data accuracy on the MNIST handwritten digits dataset. These results are even stronger as the backpropagation strategy used to train the CNNs was fairly rudimentary (ReLU units, L2 regularization and Nesterov momentum) and these were initial test runs done without refinement of the backpropagation hyperparameters. Further, the EXACT evolutionary strategy is independent of the method used to train the CNNs, so they could be further improved by advanced techniques like elastic distortions, pretraining and dropout. The evolved networks are also quite interesting, showing "organic" structures and significant differences from standard human designed architectures.
BibTeX
@inproceedings{desell2017large,
  author = {Travis Desell},
  title = {{Large Scale Evolution of Convolutional Neural Networks Using Volunteer Computing}},
  booktitle = {GECCO (Companion)},
  year = {2017},
  doi = {10.1145/3067695.3076002},
}
Figure