Abstract
Neural Architecture Search (NAS) has traditionally been computationally expensive, often requiring thousands of GPU-days to discover optimal models. This paper introduces EENA (Efficient Evolution of Neural Architecture), a framework designed to slash these costs through directional search. By utilizing function-preserving network morphisms and a novel genealogy-tracking crossover operation, EENA reuses parameters and structural knowledge from previous generations. The framework achieves a state-of-the-art 2.56% test error on CIFAR-10 in just 0.65 GPU-days, demonstrating a significant leap in search efficiency compared to traditional evolutionary and reinforcement learning approaches.
Introduction
Designing deep neural networks manually is a labor-intensive process requiring immense expertise. While Automatic Neural Architecture Search (NAS) aims to automate this design, early methods were notoriously "compute-hungry." For instance, early evolutionary and reinforcement learning (RL) approaches required hundreds or even thousands of GPU-days to find a single optimal model.
The core inefficiency in many NAS algorithms is their "directionless" nature—they treat each candidate architecture as a fresh start, training it from scratch. EENA addresses this by implementing a directional search that reuses knowledge (parameters and structures) from previously trained models, making the search process both faster and more effective.
Methodology
EENA combines evolutionary algorithms with the concept of network morphisms—transformations that change a network's architecture while preserving its underlying function. This allows the search to proceed as a continuous optimization rather than a series of disconnected experiments.
Function-Preserving Mutations
Instead of random structural changes, EENA utilizes five specific mutation types inspired by Net2Net. These include widening a layer, branching a layer, or inserting new layers (single, shortcut, or dense connections). Because these operations are function-preserving, the "offspring" network begins its life with the same performance as its "parent," eliminating the need to retrain from zero.
Guided Crossover
Standard crossover operations in NAS often destroy learned weights by blindly swapping parts of two networks. EENA solves this by tracking the "historical origins" of every mutation. When two high-performing parents are selected, EENA identifies their most recent common ancestor. It then creates an offspring that inherits the common base and randomly combines the unique mutation "genes" from both parents, ensuring that structural improvements are compounded rather than lost.
Evolutionary Strategy
The search process employs tournament selection to pick individuals for mutation and an "aging" mechanism to discard older or poorly performing models. This keeps the population size manageable while ensuring that the most fit architectures drive the evolution.
Experimental Results
CIFAR-10 Performance and Efficiency
The researchers evaluated EENA primarily on the CIFAR-10 benchmark. The search process discovered an architecture with a 2.56% test error using only 8.47M parameters. Most notably, the entire search process took only 0.65 GPU-days on a single NVIDIA Titan Xp. For comparison, AmoebaNet—a well-known evolutionary NAS—required 3,150 GPU-days to reach a similar error rate. By simply widening the channels of the discovered architecture, the error was further reduced to 2.21%.
Transferability to CIFAR-100
To test the robustness of the discovered architecture, the model optimized for CIFAR-10 was evaluated on CIFAR-100. It achieved a 17.71% error, outperforming several human-designed models like DenseNet-BC, proving that the structural traits discovered by EENA are generalizable across datasets.
Key Insights
- The "Warm Start" Advantage: Using network morphisms allows the evolution process to act as a continuous optimization of the loss function. The loss continues to decrease as the architecture grows, rather than resetting with every new candidate.
- Crossover as a Catalyst: While mutation explores the search space, crossover is significantly more effective at producing top-tier individuals. It allows the algorithm to combine successful structural traits from different lineages.
- Efficiency via Inheritance: Reusing parameters provides a "guidance of experience" that makes the search directional, achieving results comparable to gradient-based methods like DARTS but within a simpler evolutionary framework.
Conclusion and Future Work
EENA demonstrates that evolutionary NAS can be both highly efficient and effective when combined with parameter inheritance and guided crossover. While the current search space is constrained to "evolutionary blocks" within a fixed macro-structure, future work will focus on expanding the search space to include more complex units like Squeeze-and-Excitation (SE) blocks and evolving the entire macro-structure of the network.
Summary updated