FasTrCaps: An Integrated Framework for Fast yet Accurate Training of Capsule Networks

Alberto Marchisio, Beatrice Bussolino, Alessio Colucci, Muhammad Abdullah Hanif, Maurizio Martina, Guido Masera, Muhammad ShafiqueIJCNN 2020 · OpenReview

TL;DR

Capsule Networks (CapsNets) offer superior spatial relationship encoding over traditional CNNs but suffer from prohibitively long training times. This paper introduces FasTrCaps, a novel framework that integrates various optimizations, including a new WarmAdaBatch learning rate policy and weight sharing, to significantly accelerate CapsNet training while maintaining or even improving accuracy.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) have been the workhorse of deep learning for years, but they struggle to inherently capture the spatial relationships between detected features. Capsule Networks (CapsNets), introduced by Hinton et al., aim to solve this by using "capsules" – groups of neurons that encode both the presence and spatial properties of features. This allows CapsNets to be more robust to rotations and other geometric variations. However, a major roadblock to their widespread adoption is their extremely long training times, stemming from the increased complexity of their constituent elements compared to CNNs. This paper addresses the challenge of making CapsNets practical by drastically reducing their training time without sacrificing accuracy.

Methods

The core of this work is the FasTrCaps framework, which orchestrates multiple optimization techniques to accelerate CapsNet training. Key components include:

  • Tailored Learning Rate Policies: The authors adapt and evaluate standard learning rate policies (e.g., Exponential Decay, One-Cycle Policy, Warm Restarts) specifically for the CapsNet architecture, considering the unique properties of capsules and their interconnections.
  • WarmAdaBatch: This novel hybrid learning rate policy combines the benefits of Warm Restarts (for escaping local minima) and Adaptive Batch Size (for faster convergence). It dynamically adjusts both the learning rate and batch size throughout training.
  • Weight Sharing: To reduce the number of parameters, the authors propose applying weight sharing, particularly to the DigitCaps layer. This aims to decrease hardware requirements.
  • Decoder Optimization: The decoder component of CapsNets, responsible for reconstructing images, is also optimized by reducing its size, leading to fewer parameters.
  • Integrated Framework: FasTrCaps acts as an automated system that allows users to select optimization criteria (e.g., prioritizing speed, accuracy, or parameter reduction) and find Pareto-optimal solutions that balance these trade-offs.

Datasets

The framework was evaluated on two standard image classification datasets:

  • MNIST: A dataset of handwritten digits.
  • Fashion-MNIST: A dataset of grayscale fashion products, often used as a more challenging alternative to MNIST.

The authors ensured their experimental setup was compliant with previous CapsNet research, particularly the work by Google Brain [25].

Key Insights and Ideas

  • CapsNets are powerful but slow: The inherent advantage of CapsNets in preserving spatial relationships is undeniable, but their computational cost is a significant barrier.
  • Optimizations are architecture-specific: Standard deep learning optimizations don't always translate directly to CapsNets; tailoring them is crucial.
  • Warm Restarts show promise: This policy alone offered substantial training time reductions (up to 79.31%) while maintaining high accuracy.
  • Adaptive Batch Size is a double-edged sword: While it speeds up convergence, very small batch sizes can increase epoch time, making a hybrid approach more effective.
  • WarmAdaBatch is the star: This novel policy effectively combines the strengths of Warm Restarts and Adaptive Batch Size, leading to significant improvements in both speed and accuracy.
  • Weight sharing offers parameter reduction: This technique can significantly decrease model size, though it might come with a slight accuracy trade-off.
  • Pareto-optimal solutions are key: FasTrCaps doesn't just find one "best" solution; it identifies a set of optimal trade-offs, allowing users to choose based on their specific needs.

Key Results

The FasTrCaps framework demonstrated significant improvements:

  • Training Time Reduction: One solution achieved a 58.6% reduction in training time compared to the baseline CapsNet. Another achieved a 62.07% reduction using Warm Restarts.
  • Accuracy Preservation/Improvement: The framework was able to preserve accuracy and even achieve a 0.12% accuracy improvement on the MNIST dataset.
  • Parameter Reduction: Weight sharing reduced the number of parameters by more than 15% (from 8.2 million to 6.7 million), with a minimal accuracy drop.
  • Efficiency: The WarmAdaBatch policy, for instance, reached maximum accuracy in just 3 epochs compared to 29 epochs for the baseline.

Limitations and Future Work

  • Accuracy Trade-offs: While the goal is minimal accuracy loss, some optimizations, like aggressive weight sharing, can lead to a slight decrease in accuracy. The framework aims to find the best trade-offs.
  • Generalizability: While tested on MNIST and Fashion-MNIST, further evaluation on more complex datasets and tasks would be beneficial.
  • Hardware Considerations: The paper mentions potential benefits for IoT/Edge devices, but a deeper dive into hardware-specific optimizations and performance on resource-constrained platforms could be a future direction.
  • Exploration of other optimizations: The authors suggest that their framework can be extended to integrate other novel optimizations for fast training and complexity reduction.

The paper also highlights that their framework is open-sourced on GitHub, encouraging further research and development in this area.

Summary updated

Abstract

Recently, Capsule Networks (CapsNets) have shown improved performance compared to the traditional Convolutional Neural Networks (CNNs), by encoding and preserving spatial relationships between the detected features in a better way. This is achieved through the so-called Capsules (i.e., groups of neurons) that encode both the instantiation probability and the spatial information. However, one of the major hurdles in the wide adoption of CapsNets is their gigantic training time, which is primarily due to the relatively higher complexity of their new constituting elements that are different from CNNs.In this paper, we implement different optimizations in the training loop of the CapsNets, and investigate how these optimizations affect their training speed and the accuracy. Towards this, we propose a novel framework FasTrCaps that integrates multiple lightweight optimizations and a novel learning rate policy called WarmAdaBatch (that jointly performs warm restarts and adaptive batch size), and steers them in an appropriate way to provide high training-loop speedup at minimal accuracy loss. We also propose weight sharing for capsule layers. The goal is to reduce the hardware requirements of CapsNets by removing unused/redundant connections and capsules, while keeping high accuracy through tests of different learning rate policies and batch sizes. We demonstrate that one of the solutions generated by the FasTrCaps framework can achieve 58.6% reduction in the training time, while preserving the accuracy (even 0.12% accuracy improvement for the MNIST dataset), compared to the CapsNet by Google Brain [25]. Moreover, the Pareto-optimal solutions generated by FasTrCaps can be leveraged to realize trade-offs between training time and achieved accuracy. We have open-sourced our framework on GitHub <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{marchisio2020fastrcaps,
  author = {Alberto Marchisio and Beatrice Bussolino and Alessio Colucci and Muhammad Abdullah Hanif and Maurizio Martina and Guido Masera and Muhammad Shafique},
  title = {{FasTrCaps: An Integrated Framework for Fast yet Accurate Training of Capsule Networks}},
  booktitle = {IJCNN},
  year = {2020},
  url = {https://openreview.net/forum?id=qTETPNJdCAV},
}
Figure