EENA: Efficient Evolution of Neural Architecture

Hui Zhu, Zhulin An, Chuanguang Yang, Kaiqiang Xu, Yongjun Xu

Abstract

Neural Architecture Search (NAS) has traditionally been computationally expensive, often requiring thousands of GPU-days to discover optimal models. This paper introduces EENA (Efficient Evolution of Neural Architecture), a framework designed to slash these costs through directional search. By utilizing function-preserving network morphisms and a novel genealogy-tracking crossover operation, EENA reuses parameters and structural knowledge from previous generations. The framework achieves a state-of-the-art 2.56% test error on CIFAR-10 in just 0.65 GPU-days, demonstrating a significant leap in search efficiency compared to traditional evolutionary and reinforcement learning approaches.

Introduction

Designing deep neural networks manually is a labor-intensive process requiring immense expertise. While Automatic Neural Architecture Search (NAS) aims to automate this design, early methods were notoriously "compute-hungry." For instance, early evolutionary and reinforcement learning (RL) approaches required hundreds or even thousands of GPU-days to find a single optimal model.

The core inefficiency in many NAS algorithms is their "directionless" nature—they treat each candidate architecture as a fresh start, training it from scratch. EENA addresses this by implementing a directional search that reuses knowledge (parameters and structures) from previously trained models, making the search process both faster and more effective.

Methodology

EENA combines evolutionary algorithms with the concept of network morphisms—transformations that change a network's architecture while preserving its underlying function. This allows the search to proceed as a continuous optimization rather than a series of disconnected experiments.

Function-Preserving Mutations

Instead of random structural changes, EENA utilizes five specific mutation types inspired by Net2Net. These include widening a layer, branching a layer, or inserting new layers (single, shortcut, or dense connections). Because these operations are function-preserving, the "offspring" network begins its life with the same performance as its "parent," eliminating the need to retrain from zero.

Network morphism operations used in EENA to preserve function during mutation, including widening, branching, and adding connections.

Guided Crossover

Standard crossover operations in NAS often destroy learned weights by blindly swapping parts of two networks. EENA solves this by tracking the "historical origins" of every mutation. When two high-performing parents are selected, EENA identifies their most recent common ancestor. It then creates an offspring that inherits the common base and randomly combines the unique mutation "genes" from both parents, ensuring that structural improvements are compounded rather than lost.

The guided crossover mechanism identifying common ancestors to combine successful mutations from two parent architectures.

Evolutionary Strategy

The search process employs tournament selection to pick individuals for mutation and an "aging" mechanism to discard older or poorly performing models. This keeps the population size manageable while ensuring that the most fit architectures drive the evolution.

Experimental Results

CIFAR-10 Performance and Efficiency

The researchers evaluated EENA primarily on the CIFAR-10 benchmark. The search process discovered an architecture with a 2.56% test error using only 8.47M parameters. Most notably, the entire search process took only 0.65 GPU-days on a single NVIDIA Titan Xp. For comparison, AmoebaNet—a well-known evolutionary NAS—required 3,150 GPU-days to reach a similar error rate. By simply widening the channels of the discovered architecture, the error was further reduced to 2.21%.

Transferability to CIFAR-100

To test the robustness of the discovered architecture, the model optimized for CIFAR-10 was evaluated on CIFAR-100. It achieved a 17.71% error, outperforming several human-designed models like DenseNet-BC, proving that the structural traits discovered by EENA are generalizable across datasets.

Key Insights

  • The "Warm Start" Advantage: Using network morphisms allows the evolution process to act as a continuous optimization of the loss function. The loss continues to decrease as the architecture grows, rather than resetting with every new candidate.
  • Crossover as a Catalyst: While mutation explores the search space, crossover is significantly more effective at producing top-tier individuals. It allows the algorithm to combine successful structural traits from different lineages.
  • Efficiency via Inheritance: Reusing parameters provides a "guidance of experience" that makes the search directional, achieving results comparable to gradient-based methods like DARTS but within a simpler evolutionary framework.

Conclusion and Future Work

EENA demonstrates that evolutionary NAS can be both highly efficient and effective when combined with parameter inheritance and guided crossover. While the current search space is constrained to "evolutionary blocks" within a fixed macro-structure, future work will focus on expanding the search space to include more complex units like Squeeze-and-Excitation (SE) blocks and evolving the entire macro-structure of the network.

Summary updated

Abstract

Latest algorithms for automatic neural architecture search perform remarkable but are basically directionless in search space and computational expensive in training of every intermediate architecture. In this paper, we propose a method for efficient architecture search called EENA (Efficient Evolution of Neural Architecture). Due to the elaborately designed mutation and crossover operations, the evolution process can be guided by the information have already been learned. Therefore, less computational effort will be required while the searching and training time can be reduced significantly. On CIFAR-10 classification, EENA using minimal computational resources (0.65 GPU-days) can design highly effective neural architecture which achieves 2.56% test error with 8.47M parameters. Furthermore, the best architecture discovered is also transferable for CIFAR-100.
BibTeX
@inproceedings{zhu2019eena,
  author = {Hui Zhu and Zhulin An and Chuanguang Yang and Kaiqiang Xu and Yongjun Xu},
  title = {{EENA: Efficient Evolution of Neural Architecture}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00238},
}
Figure