TL;DR
NASCaps is the first Neural Architecture Search (NAS) framework designed specifically for Convolutional Capsule Networks (CapsNets). It uses a multi-objective genetic algorithm to automatically find network architectures that balance high accuracy with hardware efficiency (energy, memory, and latency) for edge deployment. By employing a clever two-stage training proxy, it drastically reduces the massive computational cost of searching through complex CapsNet configurations.
What Problem Does This Paper Address?
Capsule Networks (CapsNets) are an advanced evolution of Deep Neural Networks (DNNs). Unlike traditional Convolutional Neural Networks (CNNs) that only detect features, CapsNets encode the spatial relationships and instantiation parameters (like position and rotation) of those features. This makes them incredibly powerful for understanding hierarchical visual data.
However, this power comes at a steep cost. The capsule layers and their "dynamic routing" algorithms are notoriously compute-heavy and memory-intensive, making them difficult to deploy on resource-constrained IoT or Edge devices. Furthermore, manually designing these complex architectures is a tedious, trial-and-error process. While Neural Architecture Search (NAS) has successfully automated CNN design, existing NAS tools do not support capsule layers or dynamic routing, nor do they adequately optimize for the specific hardware constraints required to run CapsNets efficiently.
Methods
NASCaps automates the design of CapsNets using a hardware-aware, multi-objective NAS approach based on the NSGA-II genetic algorithm.
- Capsule-Aware Search Space: The authors created a modular "layer library" that includes standard convolutional layers alongside specialized capsule layers (ConvCaps, ClassCaps, and DeepCaps' CapsCells). The genetic algorithm uses crossover and mutation to explore combinations of these layers.
- Hardware Modeling: To optimize for efficiency, NASCaps integrates a high-level functional model of a specialized CapsNet hardware accelerator (CapsAcc). During the search, it estimates the latency, energy consumption, and memory footprint of every candidate network without needing full RTL simulation.
- Two-Stage Proxy Training: Training CapsNets is slow. To speed up the search, NASCaps uses a proxy evaluation. Candidate networks are trained for only a few epochs (e.g., 5 to 10). The algorithm uses these partially-trained networks to build a Pareto-front of the best models. Only the most promising architectures from this fast search are fully trained to verify their final accuracy.
Datasets
The framework was evaluated on standard image classification benchmarks:
- MNIST (Handwritten digits)
- Fashion-MNIST (FMNIST) (Clothing items)
- SVHN (Street View House Numbers)
- CIFAR-10 (10 classes of common objects/animals)
Key Insights and Ideas
- Proxy Training is Highly Reliable: The authors proved that you don't need to fully train a CapsNet to know if its architecture is good. By calculating the Pearson Correlation Coefficient (PCC), they found that just 5 epochs on MNIST or 10 epochs on CIFAR-10 yielded a >0.93 correlation with fully-trained accuracy. This insight saves massive amounts of GPU time during the NAS process.
- Capsule Logic Requires Search Constraints: To ensure the genetic algorithm produces valid and logical CapsNets, specific constraints had to be hardcoded. For example, the algorithm prevents standard convolutional layers from being placed between capsule layers, preserving the hierarchical feature-to-capsule extraction logic.
- Strong Transferability: Architectures optimized for complex datasets transfer remarkably well to simpler ones. The best architecture found during the CIFAR-10 search actually outperformed the architecture found during the MNIST-specific search when both were tested on MNIST.
Key Results
The NASCaps framework evaluated nearly 690 candidate networks using 90 GPU-hours for the search phase, yielding highly efficient Pareto-optimal models:
- CIFAR-10: The best model (
NASCaps-C10-best) achieved 85.99% accuracy. Compared to the state-of-the-art DeepCaps model, it reduced energy consumption by 52.12%, latency by 64.34%, and memory footprint by 30.19%, at the cost of only a ~1% drop in accuracy. - SVHN: Found an architecture with 92.53% accuracy that required 30.59% less energy, 59.63% less latency, and 62.70% less memory than DeepCaps.
- MNIST: The transferred
NASCaps-C10-bestmodel achieved a massive 99.72% accuracy in just 37 epochs of training.
Limitations and Future Work
- Small Population Size: The authors note that their initial random population size for the genetic algorithm was quite small (10 individuals). This likely led to a non-convergence issue in some dataset-specific searches, which explains why the CIFAR-10 optimized model outperformed the MNIST-optimized model on the MNIST dataset.
- Future Work: Because NASCaps uses a modular layer descriptor, the authors suggest that future work can easily integrate new, experimental layer types to further expand the search space. They also plan to open-source the framework to encourage the community to build upon their hardware-aware CapsNet optimizations.
Summary updated