EGFA-NAS: a neural architecture search method based on explosion gravitation field algorithm

Xuemei Hu, Lan Huang, Jia Zeng, Kangping Wang, Yan WangComplex 2023 · DOI

Abstract

Neural architecture search (NAS) is an extremely complex optimization task. Recently, population-based optimization algorithms, such as evolutionary algorithm, have been adopted as search strategies for designing neural networks automatically. Various population-based NAS methods are promising in searching for high-performance neural architectures. The explosion gravitation field algorithm (EGFA) inspired by the formation process of planets is a novel population-based optimization algorithm with excellent global optimization capability and remarkable efficiency, compared with the classical population-based algorithms, such as GA and PSO. Thus, this paper attempts to develop a more efficient NAS method, called EGFA-NAS, by utilizing the work mechanisms of EGFA, which relaxes the search discrete space to a continuous one and then utilizes EGFA and gradient descent to optimize the weights of the candidate architectures in conjunction. To reduce the computational cost, a training strategy by utilizing the population mechanism of EGFA-NAS is proposed. In addition, a weight inheritance strategy for the new generated dust individuals is proposed during the explosion operation to improve performance and efficiency. The performance of EGFA-NAS is investigated in two typical micro search spaces: NAS-Bench-201 and DARTS, and compared with various kinds of state-of-the-art NAS competitors. The experimental results demonstrate that EGFA-NAS is able to match or outperform the state-of-the-art NAS methods on image classification tasks with remarkable efficiency improvement.

Introduction

Designing high-performing deep neural networks is traditionally a manual, expert-driven process. Neural Architecture Search (NAS) aims to automate this, but it faces a massive "efficiency wall." Early NAS methods were computationally ruinous; for instance, Google’s AmoebaNet-A required 3,150 GPU days to find an optimal structure.

While newer gradient-based methods like DARTS (Differentiable Architecture Search) are faster, they often struggle with high memory usage or "premature convergence," where the search gets stuck on mediocre designs like skip-connections. This paper introduces EGFA-NAS, a method designed to maintain the global search capabilities of population-based algorithms without the prohibitive computational cost.

Methods

The authors introduce EGFA-NAS, which treats candidate neural architectures as "dust particles" in a gravitational field. The core of the approach is the Explosion Gravitation Field Algorithm (EGFA), a nature-inspired optimizer that mimics how planets form from cosmic dust.

The Explosion Gravitation Field Algorithm (EGFA) flowchart illustrating the iterative process of movement, rotation, absorption, and explosion.

Key technical innovations include:

  • Continuous Relaxation: Like DARTS, EGFA-NAS transforms the discrete choice of operations into a continuous probability map, allowing the use of gradient descent for optimization.
  • Cooperative Training Strategy: Instead of training every candidate architecture on the full dataset every epoch, the "dust individuals" cooperate. Each individual trains on a small subset of batches, but the population as a whole covers the entire dataset. This drastically reduces the per-epoch workload.
  • Weight Inheritance: When the algorithm performs an "explosion" (generating new candidate architectures around a successful "center dust" particle), the new models inherit the weights of their parent. This allows new candidates to be evaluated immediately without training from scratch.
  • Bi-level Optimization: The system simultaneously optimizes the network weights using standard gradient descent and the architecture parameters using the EGFA movement and rotation logic.
Illustration of the search process: (a) initial search space, (b) continuous relaxation, (c) optimization via EGFA, and (d) the final discrete architecture.

Results and Efficiency

The researchers evaluated EGFA-NAS on two standard benchmarks: NAS-Bench-201 and the DARTS Search Space. EGFA-NAS achieved state-of-the-art or competitive results with significantly lower search costs:

  • Speed: On the NAS-Bench-201 (CIFAR-10) task, EGFA-NAS finished its search in just 0.048 GPU days (roughly 1.15 hours). For comparison, DARTS-V2 takes 0.41 days.
  • Accuracy: On CIFAR-10, it reached 93.67% accuracy (NAS-Bench-201) and 97.23% (DARTS space). On CIFAR-100, it achieved 71.29% (NAS-Bench-201) and 81.85% (DARTS space).
  • Efficiency: In the DARTS search space, EGFA-NAS outperformed or matched almost all EA-based and RL-based competitors while using a fraction of the GPU resources (0.21 days vs. hundreds or thousands of days for older methods).

Conclusion

EGFA-NAS demonstrates that a population-based approach—usually considered the "slow and steady" option—can actually be faster than gradient-only methods if implemented with smart data-sharing and weight-sharing strategies. While the authors acknowledge the common NAS pitfall of rank consistency, future work will focus on better evaluation strategies and adapting EGFA-NAS for lightweight models targeting mobile and edge devices.

Summary updated

Abstract

Abstract Neural architecture search (NAS) is an extremely complex optimization task. Recently, population-based optimization algorithms, such as evolutionary algorithm, have been adopted as search strategies for designing neural networks automatically. Various population-based NAS methods are promising in searching for high-performance neural architectures. The explosion gravitation field algorithm (EGFA) inspired by the formation process of planets is a novel population-based optimization algorithm with excellent global optimization capability and remarkable efficiency, compared with the classical population-based algorithms, such as GA and PSO. Thus, this paper attempts to develop a more efficient NAS method, called EGFA-NAS, by utilizing the work mechanisms of EGFA, which relaxes the search discrete space to a continuous one and then utilizes EGFA and gradient descent to optimize the weights of the candidate architectures in conjunction. To reduce the computational cost, a training strategy by utilizing the population mechanism of EGFA-NAS is proposed. In addition, a weight inheritance strategy for the new generated dust individuals is proposed during the explosion operation to improve performance and efficiency. The performance of EGFA-NAS is investigated in two typical micro search spaces: NAS-Bench-201 and DARTS, and compared with various kinds of state-of-the-art NAS competitors. The experimental results demonstrate that EGFA-NAS is able to match or outperform the state-of-the-art NAS methods on image classification tasks with remarkable efficiency improvement.
BibTeX
@inproceedings{hu2023egfa,
  author = {Xuemei Hu and Lan Huang and Jia Zeng and Kangping Wang and Yan Wang},
  title = {{EGFA-NAS: a neural architecture search method based on explosion gravitation field algorithm}},
  booktitle = {Complex},
  year = {2023},
  doi = {10.1007/s40747-023-01230-0},
}
Figure