NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks

Alberto Marchisio, Andrea Massa, Vojtech Mrazek, Beatrice Bussolino, Maurizio Martina, Muhammad Shafique

Abstract

Deep Neural Networks (DNNs) have made significant improvements to reach the desired accuracy to be employed in a wide variety of Machine Learning (ML) applications. Recently the Google Brain's team demonstrated the ability of Capsule Networks (CapsNets) to encode and learn spatial correlations between different input features, thereby obtaining superior learning capabilities compared to traditional (i.e., non-capsule based) DNNs. However, designing CapsNets using conventional methods is a tedious job and incurs significant training effort. Recent studies have shown that powerful methods to automatically select the best/optimal DNN model configuration for a given set of applications and a training dataset are based on the Neural Architecture Search (NAS) algorithms. Moreover, due to their extreme computational and memory requirements, DNNs are employed using the specialized hardware accelerators in IoT-Edge/CPS devices. In this paper, we propose NASCaps, an automated framework for the hardware-aware NAS of different types of DNNs, covering both traditional convolutional DNNs and CapsNets. We study the efficacy of deploying a multi-objective Genetic Algorithm (e.g., based on the NSGA-II algorithm). The proposed framework can jointly optimize the network accuracy and the corresponding hardware efficiency, expressed in terms of energy, memory, and latency of a given hardware accelerator executing the DNN inference. Besides supporting the traditional DNN layers, our framework is the first to model and supports the specialized capsule layers and dynamic routing in the NAS-flow. We evaluate our framework on different datasets, generating different network configurations, and demonstrate the tradeoffs between the different output metrics. We will open-source the complete framework and configurations of the Pareto-optimal architectures at https://github.com/ehw-fit/nascaps.

Introduction

Capsule Networks (CapsNets) represent a significant advancement in deep learning, offering the ability to encode spatial relationships and instantiation parameters of features. This capability allows them to outperform traditional Convolutional Neural Networks (CNNs) in understanding hierarchical visual data. However, the computational intensity and memory requirements of capsule layers and their associated dynamic routing algorithms pose significant challenges for deployment on resource-constrained edge devices.

Manual design of these complex architectures is often a tedious, trial-and-error process. While Neural Architecture Search (NAS) has successfully automated the design of standard CNNs, existing frameworks lack support for capsule-specific components and hardware-aware optimization for these specialized layers. NASCaps addresses this gap by providing the first NAS framework specifically designed for Convolutional Capsule Networks.

The NASCaps Framework

NASCaps utilizes a hardware-aware, multi-objective NAS approach based on the NSGA-II genetic algorithm to automatically discover architectures that balance accuracy with hardware efficiency.

The NASCaps Framework overview, illustrating the integration of hardware modeling, layer libraries, and multi-objective optimization.

The framework consists of several key components:

  1. Capsule-Aware Search Space: A modular "layer library" includes standard convolutional layers and specialized capsule layers such as ConvCaps, ClassCaps, and CapsCells. The genetic algorithm explores this space through crossover and mutation, adhering to constraints that ensure logical hierarchical feature extraction.
  2. Hardware Modeling: To ensure efficiency, NASCaps integrates a functional model of a specialized CapsNet hardware accelerator (CapsAcc). This allows the framework to estimate latency, energy consumption, and memory footprint during the search process without requiring full RTL simulation.
  3. Two-Stage Proxy Training: To mitigate the high computational cost of training CapsNets, NASCaps employs a two-stage evaluation. Candidate networks are initially trained for a limited number of epochs (e.g., 5 to 10). The resulting partially-trained models are used to identify a Pareto-front of promising architectures, which are then fully trained to verify their final performance.
Examples of network architectures (a, b, c) and specialized layer types (d) supported within the NASCaps search space, including capsule cells and dynamic routing.

Experimental Results

The framework was evaluated on standard benchmarks including MNIST, Fashion-MNIST (FMNIST), SVHN, and CIFAR-10. The search phase evaluated nearly 690 candidate networks over 90 GPU-hours.

Proxy Training Reliability

A critical insight from this work is the reliability of proxy training. By calculating the Pearson Correlation Coefficient (PCC), the authors found that training for just 5 epochs on MNIST or 10 epochs on CIFAR-10 yielded a correlation of >0.93 with fully-trained accuracy. This significantly accelerates the NAS process.

Pareto-Optimal Architectures

NASCaps successfully identified architectures that significantly improve hardware efficiency with minimal impact on accuracy:

  • CIFAR-10: The NASCaps-C10-best model achieved 85.99% accuracy. Compared to the state-of-the-art DeepCaps model, it reduced energy consumption by 52.12%, latency by 64.34%, and memory footprint by 30.19%, with only a ~1% drop in accuracy.
  • SVHN: An architecture was found with 92.53% accuracy that required 30.59% less energy, 59.63% less latency, and 62.70% less memory than DeepCaps.
  • Transferability: Architectures optimized for complex datasets like CIFAR-10 showed strong transferability, outperforming MNIST-specific searches when tested on the MNIST dataset (achieving 99.72% accuracy).

Conclusion and Future Work

NASCaps demonstrates the feasibility of automated, hardware-aware design for complex Capsule Networks. While the initial search used a relatively small population size, the framework's modularity allows for the easy integration of new layer types in the future. The authors have open-sourced the framework to facilitate further community research into efficient CapsNet optimizations.

Summary updated

Abstract

Deep Neural Networks (DNNs) have made significant improvements to reach the desired accuracy to be employed in a wide variety of Machine Learning (ML) applications. Recently the Google Brain's team demonstrated the ability of Capsule Networks (CapsNets) to encode and learn spatial correlations between different input features, thereby obtaining superior learning capabilities compared to traditional (i.e., non-capsule based) DNNs. However, designing CapsNets using conventional methods is a tedious job and incurs significant training effort. Recent studies have shown that powerful methods to automatically select the best/optimal DNN model configuration for a given set of applications and a training dataset are based on the Neural Architecture Search (NAS) algorithms. Moreover, due to their extreme computational and memory requirements, DNNs are employed using the specialized hardware accelerators in IoT-Edge/CPS devices. In this paper, we propose NASCaps, an automated framework for the hardware-aware NAS of different types of DNNs, covering both traditional convolutional DNNs and CapsNets. We study the efficacy of deploying a multi-objective Genetic Algorithm (e.g., based on the NSGA-II algorithm). The proposed framework can jointly optimize the network accuracy and the corresponding hardware efficiency, expressed in terms of energy, memory, and latency of a given hardware accelerator executing the DNN inference. Besides supporting the traditional DNN layers, our framework is the first to model and supports the specialized capsule layers and dynamic routing in the NAS-flow. We evaluate our framework on different datasets, generating different network configurations, and demonstrate the tradeoffs between the different output metrics. We will open-source the complete framework and configurations of the Pareto-optimal architectures at https://github.com/ehw-fit/nascaps.
BibTeX
@inproceedings{marchisio2020nascaps,
  author = {Alberto Marchisio and Andrea Massa and Vojtech Mrazek and Beatrice Bussolino and Maurizio Martina and Muhammad Shafique},
  title = {{NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks}},
  booktitle = {ICCAD},
  year = {2020},
  url = {https://openreview.net/forum?id=cMtdSBBPO0B},
}
Figure