Abstract
This paper introduces Evolutionary Exploration of Augmenting Convolutional Topologies (EXACT), a novel neuroevolutionary algorithm designed to evolve the architecture of Convolutional Neural Networks (CNNs). By adapting the principles of NEAT for large-scale distributed environments, EXACT enables the discovery of high-performing, non-traditional network topologies. Utilizing the Citizen Science Grid and over 4,500 volunteer computers, the algorithm trained more than 120,000 CNNs, achieving a 98.32% accuracy on the MNIST dataset. The evolved architectures exhibit "organic" structures significantly different from human-designed models, suggesting new directions for automated neural architecture search.
Introduction
The success of deep learning, particularly Convolutional Neural Networks (CNNs), has largely depended on architectures designed by human experts through iterative trial and error. While effective, this manual design process is limited by human intuition and the immense search space of possible configurations. Neuroevolution—the use of evolutionary algorithms to design neural networks—offers a path toward automated architecture discovery. However, traditional methods like Neuroevolution of Augmenting Topologies (NEAT) struggle to scale to the complexity of CNNs and the computational demands of training them.
This research presents EXACT, an algorithm specifically engineered to evolve CNN topologies at scale. By leveraging volunteer computing via the Berkeley Open Infrastructure for Network Computing (BOINC), the study demonstrates that massive-scale neuroevolution is accessible beyond the confines of high-performance computing centers.
The EXACT Algorithm
EXACT evolves the connections between convolutional filters rather than individual neurons. It addresses the scaling challenges of NEAT through several key innovations:
Topology Evolution and Mutation
EXACT treats the network as a directed acyclic graph where nodes represent convolutional filters and edges represent connections. The algorithm employs specific mutation operators:
- Edge Modification: Enabling or disabling connections between filters.
- Node Splitting: Inserting a new filter node between existing connections.
- Node Resizing: Adjusting the dimensions of convolutional filters.
A critical mathematical observation allows any two filters to be connected by a convolution, provided the filter size is calculated based on the difference between input and output dimensions.
Asynchronous Distributed Computing
To operate within a volunteer computing environment, EXACT utilizes an asynchronous master-worker model. Unlike synchronous evolution, which requires all members of a generation to finish before proceeding, EXACT allows workers to request genomes, perform training, and report fitness independently. This handles the inherent volatility of volunteer hardware, where nodes may disconnect or vary significantly in processing speed.
Epigenetic Weight Initialization
To improve evolutionary efficiency, EXACT implements epigenetic weight initialization. Child networks can optionally inherit trained weights from their parents for unmutated parts of the topology. This "head start" allows the evolutionary process to focus on optimizing new structures rather than retraining the entire network from scratch.
Experimental Setup and Volunteer Computing
The study utilized the Citizen Science Grid, a BOINC-based project. Over a two-month period, approximately 4,500 volunteers contributed their computing power. To ensure consistency across diverse hardware and operating systems, the researchers implemented deterministic mathematical functions (e.g., exp()) and random number generators, ensuring that a model's fitness would be identical regardless of the host machine.
The MNIST dataset (60,000 training and 10,000 testing images) served as the primary benchmark. While MNIST is a standard dataset, the sheer volume of models trained (over 120,000) provided a rigorous test of the EXACT framework's scalability.
Results and Discussion
The evolutionary process yielded several significant findings:
Performance
The best-evolved networks reached a test accuracy of 98.32%. This result is particularly notable because the training regime used only basic backpropagation techniques—ReLU units, L2 regularization, and Nesterov momentum—without advanced optimizations like dropout or elastic distortions. When compared to human-designed benchmarks like LeNet-5 under the same training constraints, the EXACT-evolved networks demonstrated superior performance and efficiency.
Organic Architectures
One of the most compelling outcomes was the visual structure of the evolved networks. Unlike the sequential, layered approach favored by human designers, EXACT produced "organic" topologies. These networks featured complex branching, highly interconnected nodes, and "vestigial" structures reminiscent of biological neural pathways in the cerebral cortex.
The Role of Epigenetics
Experiments confirmed that epigenetic weight inheritance led to higher final accuracy. While it did not significantly reduce the time required for individual training runs, it allowed the evolutionary search to reach higher fitness peaks by preserving learned features across generations.
Limitations and Future Directions
While EXACT proves the viability of volunteer-driven neuroevolution, several areas for improvement remain:
- Pooling Layers: The current implementation does not evolve pooling layers, which are standard for dimensionality reduction in CNNs.
- Input Dimensionality: The system was tested on 2D grayscale images. Expanding to 3D color inputs (e.g., CIFAR-10) is a necessary next step.
- Hyperparameter Evolution: Future iterations could evolve training hyperparameters (learning rates, momentum) alongside the network topology.
Conclusion
EXACT demonstrates that evolutionary algorithms can successfully design complex CNN architectures when paired with large-scale distributed computing. By harnessing the power of thousands of volunteers, this approach democratizes neural architecture search and reveals "organic" design patterns that challenge traditional human-centric engineering.
Summary updated