DeepSwarm: Optimising Convolutional Neural Networks Using Swarm Intelligence

Edvinas Byla, Wei Pang

TL;DR

DeepSwarm is a novel Neural Architecture Search (NAS) method that uses Ant Colony Optimization to automatically design high-performing Convolutional Neural Networks (CNNs). By mimicking the collective foraging behavior of ants, it efficiently navigates the massive space of possible network designs to find optimal architectures with competitive accuracy and lower computational costs.

What Problem Does This Paper Address?

Designing a deep neural network is often more of an art than a science. Human engineers spend countless hours tuning hyperparameters and stacking layers, only to find that an architecture that works for one dataset (like identifying digits) fails miserably on another (like identifying objects in photos).

Neural Architecture Search (NAS) aims to automate this design process. However, early NAS methods—often based on Reinforcement Learning or Evolutionary Algorithms—required staggering amounts of computational power (think thousands of GPU hours). The authors of DeepSwarm tackle the challenge of making NAS more efficient, flexible, and accessible to researchers who don't have a Google-sized server farm.

Methods

The researchers turned to Swarm Intelligence, specifically Ant Colony Optimization (ACO). Here is how the "digital ants" build a brain:

  1. The Graph Search: The search space is represented as a graph where nodes are neural network layers (Conv2D, Pooling, etc.). Ants traverse this graph to create a "path," which represents a complete network architecture.
  2. Pheromone Trails: Just like real ants leave chemical trails to food, DeepSwarm ants leave "pheromones" on successful architecture paths.
  3. Local Update: When an ant finishes a path, it slightly reduces the pheromone on that trail to encourage other ants to explore different, potentially better paths (preventing "groupthink").
  4. Global Update: Once all ants in a "generation" are evaluated, the best-performing ant's path is reinforced with extra pheromones, guiding future ants toward high-accuracy designs.
  5. Progressive Search: Instead of trying to build a 100-layer network immediately, DeepSwarm starts with simple architectures and incrementally increases the maximum allowed depth.
  6. Weight Reusability: To save time, DeepSwarm doesn't always train from scratch. If a new architecture shares a "sub-path" with a previously trained one, it reuses those weights, drastically speeding up the evaluation process.
  7. Heuristic Integration: Unlike many "black-box" NAS methods, DeepSwarm allows human experts to inject "heuristic information" (prior knowledge) to guide the ants toward choices known to be generally effective.

Datasets

The authors evaluated DeepSwarm on three standard computer vision benchmarks:

  • MNIST: The classic dataset of handwritten digits.
  • Fashion-MNIST: A more challenging drop-in replacement for MNIST featuring clothing items.
  • CIFAR-10: 60,000 color images in 10 classes (animals, vehicles, etc.), representing a significant jump in complexity.

Key Insights and Ideas

  • The "Goldilocks" Greediness: The authors found that setting the "greediness" parameter to 0.5 (the middle ground) yielded the best results. If ants are too greedy, they converge on mediocre designs too quickly; if they aren't greedy enough, the search becomes a random, inefficient mess.
  • Diminishing Returns of the Swarm: Increasing the number of ants improves accuracy, but only up to a point. Moving from 4 to 8 ants more than doubled the runtime (7 hours to 18 hours) but only improved the error rate by a tiny 0.13%.
  • Generalization vs. Complexity: Less greedy ants tended to find deeper architectures. Interestingly, these deeper, more complex models often generalized better to new data after final training, likely due to better feature extraction.

Key Results

DeepSwarm demonstrated competitive performance across the board, often outperforming standard baselines like random or grid search:

Dataset DeepSwarm (Best Error Rate) Comparison
MNIST 0.39% Better than Random (1.79%) and Grid Search (1.68%)
Fashion-MNIST 6.44% Significantly better than AutoKeras BFS (7.42%)
CIFAR-10 11.31% Competitive with NASBOT (12.30%) and SEAS (12.43%)

While Google’s massive NAS systems (using 400 GPUs) can achieve slightly better results on CIFAR-10, DeepSwarm achieved its results using a single NVIDIA Tesla K80, making it far more efficient for the average practitioner.

Limitations and Future Work

The authors identify a few areas where the "ants" could improve:

  • Architectural Constraints: The current version of DeepSwarm primarily searches for sequential architectures. It does not yet support "skip connections" or "add nodes" (like those found in ResNet), which are crucial for very deep networks.
  • Cell-Based Search: Future work could involve using ACO to design small "cells" or blocks that are then repeated, a strategy that has proven successful in other state-of-the-art NAS methods.
  • Dropout Optimization: The authors suggest using ACO to determine exactly which neurons to "drop out" during training to further improve regularization.

DeepSwarm is available as an open-source library on GitHub.

Summary updated

Abstract

In this paper we propose DeepSwarm, a novel neural architecture search (NAS) method based on Swarm Intelligence principles. At its core DeepSwarm uses Ant Colony Optimization (ACO) to generate ant population which uses the pheromone information to collectively search for the best neural architecture. Furthermore, by using local and global pheromone update rules our method ensures the balance between exploitation and exploration. On top of this, to make our method more efficient we combine progressive neural architecture search with weight reusability. Furthermore, due to the nature of ACO our method can incorporate heuristic information which can further speed up the search process. After systematic and extensive evaluation, we discover that on three different datasets (MNIST, Fashion-MNIST, and CIFAR-10) when compared to existing systems our proposed method demonstrates competitive performance. Finally, we open source DeepSwarm (https://github.com/Pattio/DeepSwarm) as a NAS library and hope it can be used by more deep learning researchers and practitioners.
BibTeX
@inproceedings{byla2019deepswarm,
  author = {Edvinas Byla and Wei Pang},
  title = {{DeepSwarm: Optimising Convolutional Neural Networks Using Swarm Intelligence}},
  booktitle = {UKCI},
  year = {2019},
  doi = {10.1007/978-3-030-29933-0\_10},
}

Loading related content...

Figure