NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks

Alberto Marchisio, Andrea Massa, Vojtech Mrazek, Beatrice Bussolino, Maurizio Martina, Muhammad Shafique

TL;DR

NASCaps is the first Neural Architecture Search (NAS) framework designed specifically for Convolutional Capsule Networks (CapsNets). It uses a multi-objective genetic algorithm to automatically find network architectures that balance high accuracy with hardware efficiency (energy, memory, and latency) for edge deployment. By employing a clever two-stage training proxy, it drastically reduces the massive computational cost of searching through complex CapsNet configurations.

What Problem Does This Paper Address?

Capsule Networks (CapsNets) are an advanced evolution of Deep Neural Networks (DNNs). Unlike traditional Convolutional Neural Networks (CNNs) that only detect features, CapsNets encode the spatial relationships and instantiation parameters (like position and rotation) of those features. This makes them incredibly powerful for understanding hierarchical visual data.

However, this power comes at a steep cost. The capsule layers and their "dynamic routing" algorithms are notoriously compute-heavy and memory-intensive, making them difficult to deploy on resource-constrained IoT or Edge devices. Furthermore, manually designing these complex architectures is a tedious, trial-and-error process. While Neural Architecture Search (NAS) has successfully automated CNN design, existing NAS tools do not support capsule layers or dynamic routing, nor do they adequately optimize for the specific hardware constraints required to run CapsNets efficiently.

Methods

NASCaps automates the design of CapsNets using a hardware-aware, multi-objective NAS approach based on the NSGA-II genetic algorithm.

  • Capsule-Aware Search Space: The authors created a modular "layer library" that includes standard convolutional layers alongside specialized capsule layers (ConvCaps, ClassCaps, and DeepCaps' CapsCells). The genetic algorithm uses crossover and mutation to explore combinations of these layers.
  • Hardware Modeling: To optimize for efficiency, NASCaps integrates a high-level functional model of a specialized CapsNet hardware accelerator (CapsAcc). During the search, it estimates the latency, energy consumption, and memory footprint of every candidate network without needing full RTL simulation.
  • Two-Stage Proxy Training: Training CapsNets is slow. To speed up the search, NASCaps uses a proxy evaluation. Candidate networks are trained for only a few epochs (e.g., 5 to 10). The algorithm uses these partially-trained networks to build a Pareto-front of the best models. Only the most promising architectures from this fast search are fully trained to verify their final accuracy.

Datasets

The framework was evaluated on standard image classification benchmarks:

  • MNIST (Handwritten digits)
  • Fashion-MNIST (FMNIST) (Clothing items)
  • SVHN (Street View House Numbers)
  • CIFAR-10 (10 classes of common objects/animals)

Key Insights and Ideas

  • Proxy Training is Highly Reliable: The authors proved that you don't need to fully train a CapsNet to know if its architecture is good. By calculating the Pearson Correlation Coefficient (PCC), they found that just 5 epochs on MNIST or 10 epochs on CIFAR-10 yielded a >0.93 correlation with fully-trained accuracy. This insight saves massive amounts of GPU time during the NAS process.
  • Capsule Logic Requires Search Constraints: To ensure the genetic algorithm produces valid and logical CapsNets, specific constraints had to be hardcoded. For example, the algorithm prevents standard convolutional layers from being placed between capsule layers, preserving the hierarchical feature-to-capsule extraction logic.
  • Strong Transferability: Architectures optimized for complex datasets transfer remarkably well to simpler ones. The best architecture found during the CIFAR-10 search actually outperformed the architecture found during the MNIST-specific search when both were tested on MNIST.

Key Results

The NASCaps framework evaluated nearly 690 candidate networks using 90 GPU-hours for the search phase, yielding highly efficient Pareto-optimal models:

  • CIFAR-10: The best model (NASCaps-C10-best) achieved 85.99% accuracy. Compared to the state-of-the-art DeepCaps model, it reduced energy consumption by 52.12%, latency by 64.34%, and memory footprint by 30.19%, at the cost of only a ~1% drop in accuracy.
  • SVHN: Found an architecture with 92.53% accuracy that required 30.59% less energy, 59.63% less latency, and 62.70% less memory than DeepCaps.
  • MNIST: The transferred NASCaps-C10-best model achieved a massive 99.72% accuracy in just 37 epochs of training.

Limitations and Future Work

  • Small Population Size: The authors note that their initial random population size for the genetic algorithm was quite small (10 individuals). This likely led to a non-convergence issue in some dataset-specific searches, which explains why the CIFAR-10 optimized model outperformed the MNIST-optimized model on the MNIST dataset.
  • Future Work: Because NASCaps uses a modular layer descriptor, the authors suggest that future work can easily integrate new, experimental layer types to further expand the search space. They also plan to open-source the framework to encourage the community to build upon their hardware-aware CapsNet optimizations.

Summary updated

Abstract

Deep Neural Networks (DNNs) have made significant improvements to reach the desired accuracy to be employed in a wide variety of Machine Learning (ML) applications. Recently the Google Brain's team demonstrated the ability of Capsule Networks (CapsNets) to encode and learn spatial correlations between different input features, thereby obtaining superior learning capabilities compared to traditional (i.e., non-capsule based) DNNs. However, designing CapsNets using conventional methods is a tedious job and incurs significant training effort. Recent studies have shown that powerful methods to automatically select the best/optimal DNN model configuration for a given set of applications and a training dataset are based on the Neural Architecture Search (NAS) algorithms. Moreover, due to their extreme computational and memory requirements, DNNs are employed using the specialized hardware accelerators in IoT-Edge/CPS devices. In this paper, we propose NASCaps, an automated framework for the hardware-aware NAS of different types of DNNs, covering both traditional convolutional DNNs and CapsNets. We study the efficacy of deploying a multi-objective Genetic Algorithm (e.g., based on the NSGA-II algorithm). The proposed framework can jointly optimize the network accuracy and the corresponding hardware efficiency, expressed in terms of energy, memory, and latency of a given hardware accelerator executing the DNN inference. Besides supporting the traditional DNN layers (such as, convolutional and fully-connected), our framework is the first to model and supports the specialized capsule layers and dynamic routing in the NAS-flow. We evaluate our framework on different datasets, generating different network configurations, and demonstrate the tradeoffs between the different output metrics. We will open-source the complete framework and configurations of the Pareto-optimal architectures at https://github.com/ehw-fit/nascaps.
BibTeX
@inproceedings{marchisio2020nascaps,
  author = {Alberto Marchisio and Andrea Massa and Vojtech Mrazek and Beatrice Bussolino and Maurizio Martina and Muhammad Shafique},
  title = {{NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks}},
  booktitle = {ICCAD},
  year = {2020},
  url = {https://openreview.net/forum?id=cMtdSBBPO0B},
}

Loading related content...

Figure