Abstract
Automated architecture search for Convolutional Neural Networks (CNNs) offers a path toward discovering high-performing models without the need for manual expert design. However, traditional neuro-evolutionary methods often struggle with the high computational costs and slow convergence rates associated with deep learning. This paper presents the EXACT (Evolutionary eXploration of Augmenting Convolutional Topologies) algorithm, which introduces three key strategies to accelerate this process: node-level mutations, epigenetic weight initialization, and fractional max pooling. By leveraging a massive volunteer computing grid to evaluate over 225,000 architectures, we demonstrate that these enhancements significantly improve the efficiency and accuracy of evolved networks, achieving a peak accuracy of 99.46% on the MNIST dataset.
Introduction
The design of high-performing Convolutional Neural Networks (CNNs) has historically been a manual, expert-driven process. While architectures like ResNet and VGGNet have set benchmarks for performance, they represent a narrow slice of the potential design space. Neuro-evolution aims to automate this discovery by "breeding" network architectures through iterative mutation and selection.
Despite its potential, neuro-evolution is often hindered by the "micro-mutation" bottleneck. Traditional algorithms, such as NEAT (NeuroEvolution of Augmenting Topologies), focus on incremental changes—such as adding a single connection between neurons. For the complex, multi-dimensional feature maps used in CNNs, these tiny steps are often too slow to find optimal structures within a reasonable timeframe. This research addresses these limitations by introducing "macro" operations and smarter initialization techniques to make neuro-evolution a viable tool for deep learning.
Methodology
The EXACT Algorithm
The Evolutionary eXploration of Augmenting Convolutional Topologies (EXACT) algorithm evolves the structure of feature maps and the filters that connect them. To improve the speed and efficacy of the search, three specific enhancements were implemented:
- Node-Level Mutations: Rather than limiting mutations to individual connections (edges), EXACT utilizes operations that add, split, or merge entire feature maps (nodes). This allows the network to increase in complexity through significant structural leaps, rather than incremental adjustments.
- Epigenetic Weight Initialization: In standard evolutionary algorithms, child networks typically begin with randomized weights. EXACT employs "epigenetic" initialization, where child networks inherit the trained weights of their parents. This allows the evolutionary process to preserve and build upon the "learned experience" of previous generations.
- Fractional Max Pooling: To facilitate connections between feature maps of arbitrary sizes without the constraints of standard integer pooling ratios (e.g., 2x2), the algorithm utilizes fractional max pooling. This technique uses randomized pooling regions to handle non-standard dimensions, ensuring structural flexibility.
Distributed Computing Infrastructure
To manage the immense computational requirements of evaluating hundreds of thousands of architectures, the study utilized the Citizen Science Grid. This volunteer computing platform, powered by BOINC, harnessed over 13,000 CPUs globally, enabling a massive parallel search that would be prohibitive on standard local clusters.
Experimental Setup
The benchmark for this study was the MNIST dataset of handwritten digits. The data was partitioned into 50,000 images for training, 10,000 for a validation set (used to determine fitness during the evolutionary process), and a separate 10,000-image test set for final evaluation. This rigorous separation ensured that the evolved networks were capable of generalization rather than simply memorizing the training data.
Results and Discussion
Performance and Efficiency
The best-evolved CNN reached a top accuracy of 99.46% on the MNIST test set, a result comparable to state-of-the-art human-designed networks. The inclusion of node-level mutations was particularly impactful, reducing test error by approximately 0.2% compared to traditional edge-only mutations. This suggests that for complex tasks, the ability to make "macro" structural changes is essential for navigating the architecture search space effectively.
The Impact of Weight Inheritance
Epigenetic weight initialization proved to be a critical factor in both speed and final performance. Inheriting weights reduced test error rates by 0.11% to 0.44% (with an average improvement of 0.62%) compared to random initialization. This indicates that "nature vs. nurture" is not a binary in neuro-evolution; providing a child network with a pre-trained starting point allows the algorithm to navigate the weight landscape more effectively.
The Pooling Paradox
An unexpected finding was that allowing the evolution to utilize pooling connections actually degraded performance. The researchers hypothesize that adding pooling options expanded the search space to such a degree that it became harder for the algorithm to identify beneficial mutations. This highlights a recurring challenge in automated design: increasing the degrees of freedom can sometimes lead to less efficient optimization.
Structural Observations
The evolved networks exhibited an "organic" complexity. Unlike the neat, sequential layers found in human-designed models, the EXACT-generated networks were highly interconnected webs featuring numerous skip-connections. This suggests that optimal architectures for specific tasks may be far more irregular and non-linear than human intuition typically suggests.
Conclusion
This study demonstrates that node-level mutations and epigenetic weight initialization can significantly accelerate the evolution of high-performing CNNs. By evaluating over 225,000 architectures via volunteer computing, the EXACT algorithm proved capable of discovering complex, irregular structures that rival human designs.
Future work will focus on scaling these strategies to more complex datasets like CIFAR-10 and ImageNet. Additionally, exploring online adaptation—where mutation rates change dynamically during the search—and multiple simultaneous mutations may further enhance the discovery of sophisticated neural architectures.
Summary updated