Capsule Network based Contrastive Learning of Unsupervised Visual Representations

Harsh Panwar, Ioannis PatrasarXiv 2022 · DOI · arXiv

Abstract

This paper introduces Contrastive Capsule (CoCa), a novel framework that integrates Capsule Networks with Siamese-style contrastive learning for unsupervised visual representation. By utilizing the equivariant properties of capsules, CoCa achieves competitive performance on the CIFAR-10 dataset while significantly reducing computational overhead. The proposed model requires 31x fewer parameters and 71x fewer FLOPs than state-of-the-art CNN-based models like SimCLR, demonstrating a highly efficient path toward learning meaningful visual features without human supervision.

Introduction

Traditional Convolutional Neural Networks (CNNs) have long dominated visual tasks but suffer from a fundamental "spatial" limitation. While excellent at detecting the presence of features, CNNs often struggle to maintain precise spatial relationships (position, size, orientation) between those features. This is largely due to max-pooling layers, which provide translation invariance at the cost of discarding detailed positional information.

Capsule Networks (CapsNets) were designed to address this by replacing scalar neurons with vectors. These vectors represent the "pose" of an object part, allowing the network to naturally track transformations—a property known as equivariance.

Comparison of operational differences between Capsule Networks and standard Neural Networks.

While most CapsNet research has focused on supervised learning, the CoCa model explores their potential in unsupervised learning, where the goal is to learn representations from raw pixels by maximizing agreement between differently augmented views of the same image.

The CoCa Framework

The Contrastive Capsule (CoCa) model utilizes a Siamese network architecture consisting of two identical branches. Each branch processes a different augmented version of the same input image to learn invariant and equivariant features.

Architecture Components

  1. Encoder: A shallow 6-layer convolutional block (ConvBlock) acts as the initial feature extractor.
  2. PrimaryCaps: This layer converts the convolutional features into 16-dimensional vectors (capsules).
  3. ClassCaps: Using a "routing by agreement" mechanism, the model groups lower-level features into higher-level representations. This dynamic routing provides a form of top-down feedback, helping the model determine part-whole relationships during the forward pass.
  4. Contrastive Loss (NT-Xent): The model is trained to maximize the similarity between representations of the same image (positives) while minimizing similarity with other images in the batch (negatives).
The Siamese architecture of the Contrastive Capsule (CoCa) model.

Experimental Results

The model was evaluated on the CIFAR-10 dataset, consisting of 60,000 images across 10 classes. To evaluate the learned representations, a k-Nearest Neighbor (k-NN) approach was used, comparing test image features against a memory bank of training features.

Performance and Efficiency

  • Accuracy: CoCa achieved a 70.50% top-1 accuracy and 98.10% top-5 accuracy.
  • Parameter Efficiency: The model utilizes only 780,000 parameters, a 31x reduction compared to the 24.62 million parameters in SimCLR (ResNet-50).
  • Computational Speed: CoCa requires 71x fewer FLOPs and trains 4x faster on identical hardware compared to current state-of-the-art unsupervised CNNs.

These results suggest that the inherent equivariance of capsules acts as a "shortcut," allowing the model to learn spatial properties that CNNs must learn through massive data augmentation and deeper architectures.

Conclusion and Future Work

CoCa demonstrates that Capsule Networks are highly effective for unsupervised representation learning, offering massive gains in computational efficiency. Future research will focus on addressing routing bottlenecks, such as the computational expense of dynamic routing, and scaling the model with larger batch sizes and advanced optimization techniques like LARS or Gaussian blur. The authors suggest that CoCa's spatial precision makes it particularly suitable for domains where labeled data is scarce but geometric accuracy is critical, such as medical imaging.

Summary updated

Abstract

Capsule Networks have shown tremendous advancement in the past decade, outperforming the traditional CNNs in various task due to it's equivariant properties. With the use of vector I/O which provides information of both magnitude and direction of an object or it's part, there lies an enormous possibility of using Capsule Networks in unsupervised learning environment for visual representation tasks such as multi class image classification. In this paper, we propose Contrastive Capsule (CoCa) Model which is a Siamese style Capsule Network using Contrastive loss with our novel architecture, training and testing algorithm. We evaluate the model on unsupervised image classification CIFAR-10 dataset and achieve a top-1 test accuracy of 70.50% and top-5 test accuracy of 98.10%. Due to our efficient architecture our model has 31 times less parameters and 71 times less FLOPs than the current SOTA in both supervised and unsupervised learning.
BibTeX
@misc{panwar2022capsule,
  author = {Harsh Panwar and Ioannis Patras},
  title = {{Capsule Network based Contrastive Learning of Unsupervised Visual Representations}},
  year = {2022},
  eprint = {2209.11276},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.11276},
}
Figure