Abstract
Neural architecture search (NAS) is an extremely complex optimization task. Recently, population-based optimization algorithms, such as evolutionary algorithm, have been adopted as search strategies for designing neural networks automatically. Various population-based NAS methods are promising in searching for high-performance neural architectures. The explosion gravitation field algorithm (EGFA) inspired by the formation process of planets is a novel population-based optimization algorithm with excellent global optimization capability and remarkable efficiency, compared with the classical population-based algorithms, such as GA and PSO. Thus, this paper attempts to develop a more efficient NAS method, called EGFA-NAS, by utilizing the work mechanisms of EGFA, which relaxes the search discrete space to a continuous one and then utilizes EGFA and gradient descent to optimize the weights of the candidate architectures in conjunction. To reduce the computational cost, a training strategy by utilizing the population mechanism of EGFA-NAS is proposed. In addition, a weight inheritance strategy for the new generated dust individuals is proposed during the explosion operation to improve performance and efficiency. The performance of EGFA-NAS is investigated in two typical micro search spaces: NAS-Bench-201 and DARTS, and compared with various kinds of state-of-the-art NAS competitors. The experimental results demonstrate that EGFA-NAS is able to match or outperform the state-of-the-art NAS methods on image classification tasks with remarkable efficiency improvement.
Introduction
Designing high-performing deep neural networks is traditionally a manual, expert-driven process. Neural Architecture Search (NAS) aims to automate this, but it faces a massive "efficiency wall." Early NAS methods were computationally ruinous; for instance, Google’s AmoebaNet-A required 3,150 GPU days to find an optimal structure.
While newer gradient-based methods like DARTS (Differentiable Architecture Search) are faster, they often struggle with high memory usage or "premature convergence," where the search gets stuck on mediocre designs like skip-connections. This paper introduces EGFA-NAS, a method designed to maintain the global search capabilities of population-based algorithms without the prohibitive computational cost.
Methods
The authors introduce EGFA-NAS, which treats candidate neural architectures as "dust particles" in a gravitational field. The core of the approach is the Explosion Gravitation Field Algorithm (EGFA), a nature-inspired optimizer that mimics how planets form from cosmic dust.
Key technical innovations include:
- Continuous Relaxation: Like DARTS, EGFA-NAS transforms the discrete choice of operations into a continuous probability map, allowing the use of gradient descent for optimization.
- Cooperative Training Strategy: Instead of training every candidate architecture on the full dataset every epoch, the "dust individuals" cooperate. Each individual trains on a small subset of batches, but the population as a whole covers the entire dataset. This drastically reduces the per-epoch workload.
- Weight Inheritance: When the algorithm performs an "explosion" (generating new candidate architectures around a successful "center dust" particle), the new models inherit the weights of their parent. This allows new candidates to be evaluated immediately without training from scratch.
- Bi-level Optimization: The system simultaneously optimizes the network weights using standard gradient descent and the architecture parameters using the EGFA movement and rotation logic.
Results and Efficiency
The researchers evaluated EGFA-NAS on two standard benchmarks: NAS-Bench-201 and the DARTS Search Space. EGFA-NAS achieved state-of-the-art or competitive results with significantly lower search costs:
- Speed: On the NAS-Bench-201 (CIFAR-10) task, EGFA-NAS finished its search in just 0.048 GPU days (roughly 1.15 hours). For comparison, DARTS-V2 takes 0.41 days.
- Accuracy: On CIFAR-10, it reached 93.67% accuracy (NAS-Bench-201) and 97.23% (DARTS space). On CIFAR-100, it achieved 71.29% (NAS-Bench-201) and 81.85% (DARTS space).
- Efficiency: In the DARTS search space, EGFA-NAS outperformed or matched almost all EA-based and RL-based competitors while using a fraction of the GPU resources (0.21 days vs. hundreds or thousands of days for older methods).
Conclusion
EGFA-NAS demonstrates that a population-based approach—usually considered the "slow and steady" option—can actually be faster than gradient-only methods if implemented with smart data-sharing and weight-sharing strategies. While the authors acknowledge the common NAS pitfall of rank consistency, future work will focus on better evaluation strategies and adapting EGFA-NAS for lightweight models targeting mobile and edge devices.
Summary updated