EG-NAS: Neural Architecture Search with Fast Evolutionary Exploration

Zicheng Cai, Lei Chen, Peng Liu, Tongtao Ling, Yutao LaiAAAI 2024 · DOI

TL;DR

This paper introduces EG-NAS, a novel Neural Architecture Search (NAS) method that combines gradient descent with an evolutionary strategy to overcome the common issue of premature convergence in NAS. By intelligently balancing exploration and exploitation, EG-NAS achieves state-of-the-art performance with significantly reduced search costs.

What Problem Does This Paper Address?

The field of deep learning has seen tremendous progress driven by sophisticated neural network architectures. However, manually designing these architectures is a time-consuming and expensive process. Neural Architecture Search (NAS) automates this, but existing methods face challenges. Differentiable Architecture Search (DARTS) offers efficiency by using gradient descent, but it often converges to suboptimal solutions (local optima). Traditional evolutionary algorithms (EA) can explore more broadly but are computationally expensive. The core problem this paper addresses is the trade-off between search efficiency and the risk of premature convergence to local optima in NAS.

Methods

EG-NAS proposes a hybrid approach to NAS. It leverages the efficiency of gradient descent for updating network parameters, similar to DARTS. However, to mitigate premature convergence, it introduces an evolutionary strategy (specifically inspired by CMA-ES) to optimize the architecture parameters. This evolutionary component explores a wider range of search directions.

A key innovation is the compound fitness function used in the evolutionary strategy. Instead of solely relying on accuracy, it incorporates both the cross-entropy loss (representing performance) and a measure of individual similarity. This dual focus allows the algorithm to:

  • Promote diversity: When a new architecture performs better, the fitness function encourages exploration of new directions.
  • Guide convergence: When a new architecture is worse, it steers the search back towards promising directions.

The overall EG-NAS framework iteratively updates network weights using gradient descent and then uses the evolutionary strategy to find a new, promising search direction for the architecture parameters.

Datasets

The paper evaluates EG-NAS on the following datasets:

  • CIFAR-10: A standard benchmark for image classification.
  • CIFAR-100: A more challenging version of CIFAR-10 with more classes.
  • ImageNet: A large-scale dataset for image classification.
  • NAS-Bench-201: A benchmark specifically designed for NAS research, offering a standardized search space and evaluation framework across CIFAR-10, CIFAR-100, and ImageNet16-120.

Key Insights and Ideas

  • Hybridization is key: Combining the strengths of gradient-based methods (efficiency) and evolutionary algorithms (global exploration) is a powerful strategy for NAS.
  • Beyond pure accuracy: Redefining the fitness function to include individual similarity is crucial for guiding the evolutionary search towards diverse and high-performing architectures, preventing it from getting stuck.
  • Targeted exploration: The compound fitness function intelligently balances exploring new architectural possibilities with converging towards good solutions, effectively addressing the local optima problem.
  • Efficiency gains are substantial: The proposed method achieves competitive results with drastically reduced search times compared to many existing NAS techniques.

Key Results

  • On CIFAR-10, EG-NAS achieves 97.47% accuracy with a search cost of just 0.1 GPU-Days.
  • When transferred to ImageNet, the CIFAR-10 searched architecture achieves 74.4% top-1 accuracy.
  • Directly searching on ImageNet, EG-NAS finds an architecture with 75.1% top-1 accuracy in only 1.2 GPU-Days (using 2 RTX 4090 GPUs). This is highlighted as an optimal search speed.
  • On NAS-Bench-201, EG-NAS achieves competitive results across CIFAR-10, CIFAR-100, and ImageNet16-120, demonstrating its generalizability.

Limitations and Future Work

  • Ablation Study on Fitness Function: The paper includes an ablation study (Figure 3) showing that while fitness functions solely based on performance are good initially, the combined fitness function (L1 + L2) shows superior growth and performance over time. Fitness functions focusing solely on diversity struggle with convergence.
  • Population Size Impact: Experiments (Figure 4) show that a small population size limits diversity, while a very large population size offers diminishing returns and increases search cost. The authors settled on a population size of 50 for their experiments.
  • Future Work: The authors aim to further investigate the effective integration of different algorithms and enhance the stability of hybrid algorithms to address a wider range of tasks.

Summary updated

BibTeX
@inproceedings{cai2024eg,
  author = {Zicheng Cai and Lei Chen and Peng Liu and Tongtao Ling and Yutao Lai},
  title = {{EG-NAS: Neural Architecture Search with Fast Evolutionary Exploration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i10.28993},
}
Figure