Abstract
In this paper we propose DeepSwarm, a novel neural architecture search (NAS) method based on Swarm Intelligence principles. At its core DeepSwarm uses Ant Colony Optimization (ACO) to generate ant population which uses the pheromone information to collectively search for the best neural architecture. Furthermore, by using local and global pheromone update rules our method ensures the balance between exploitation and exploration. On top of this, to make our method more efficient we combine progressive neural architecture search with weight reusability. Furthermore, due to the nature of ACO our method can incorporate heuristic information which can further speed up the search process. After systematic and extensive evaluation, we discover that on three different datasets (MNIST, Fashion-MNIST, and CIFAR-10) when compared to existing systems our proposed method demonstrates competitive performance. Finally, we open source DeepSwarm as a NAS library and hope it can be used by more deep learning researchers and practitioners.
Introduction
The design of deep neural networks has traditionally relied on human expertise, involving extensive hyperparameter tuning and architectural experimentation. Neural Architecture Search (NAS) emerged to automate this process, yet early approaches—primarily based on Reinforcement Learning or Evolutionary Algorithms—demanded prohibitive computational resources. DeepSwarm addresses this efficiency gap by leveraging Ant Colony Optimization (ACO) to navigate the vast search space of network architectures more effectively, making automated design accessible without requiring massive server farms.
Methodology: The DeepSwarm Algorithm
DeepSwarm represents the search space as a directed graph where nodes correspond to neural network layers, such as Conv2D or Pooling layers. The search process is driven by "digital ants" that traverse this graph to construct complete network paths.
The algorithm incorporates several key mechanisms:
- Pheromone Trails: Ants deposit pheromones on successful paths. A local update rule reduces pheromones on a chosen trail to encourage exploration, while a global update reinforces the best-performing path of a generation to guide future ants toward optimal designs.
- Progressive Search: The system starts with simple, shallow architectures and incrementally increases the maximum allowed depth, focusing search efforts on promising foundations.
- Weight Reusability: To minimize training time, DeepSwarm reuses weights from previously evaluated sub-paths, significantly accelerating the evaluation of new architectures.
- Heuristic Integration: Human prior knowledge can be injected as heuristic information to bias the ants' choices toward architectural patterns known to be effective.
Experimental Evaluation
The performance of DeepSwarm was evaluated on three standard benchmarks: MNIST (handwritten digits), Fashion-MNIST (clothing items), and CIFAR-10 (complex color images).
Key Insights
A critical hyperparameter in DeepSwarm is "greediness," which controls the balance between following established pheromone trails (exploitation) and exploring new paths.
Experimental results indicated that a greediness of 0.5 provided the optimal balance. High greediness led to premature convergence on sub-optimal designs, while low greediness resulted in an inefficient, near-random search. Additionally, while increasing the ant count generally improved accuracy, it also introduced significant computational overhead with diminishing returns; for instance, doubling the swarm size from 4 to 8 ants increased runtime from 7 to 18 hours for a marginal 0.13% improvement in error rate.
Results
DeepSwarm demonstrated competitive performance against established baselines and other NAS methods:
| Dataset | DeepSwarm Error Rate | Comparison Baseline |
|---|---|---|
| MNIST | 0.39% | Random Search (1.79%) |
| Fashion-MNIST | 6.44% | AutoKeras BFS (7.42%) |
| CIFAR-10 | 11.31% | NASBOT (12.30%) |
Notably, DeepSwarm achieved these results using a single NVIDIA Tesla K80, highlighting its efficiency compared to large-scale systems that utilize hundreds of GPUs.
Conclusion and Future Work
DeepSwarm provides a robust, swarm-intelligence-based framework for automated neural architecture design. While effective for sequential architectures, future development will focus on supporting non-sequential structures like skip connections (e.g., ResNet) and cell-based search strategies. The authors have released DeepSwarm as an open-source library to facilitate further research in efficient NAS.
Summary updated