Abstract
This paper introduces Contrastive Capsule (CoCa), a novel framework that integrates Capsule Networks with Siamese-style contrastive learning for unsupervised visual representation. By utilizing the equivariant properties of capsules, CoCa achieves competitive performance on the CIFAR-10 dataset while significantly reducing computational overhead. The proposed model requires 31x fewer parameters and 71x fewer FLOPs than state-of-the-art CNN-based models like SimCLR, demonstrating a highly efficient path toward learning meaningful visual features without human supervision.
Introduction
Traditional Convolutional Neural Networks (CNNs) have long dominated visual tasks but suffer from a fundamental "spatial" limitation. While excellent at detecting the presence of features, CNNs often struggle to maintain precise spatial relationships (position, size, orientation) between those features. This is largely due to max-pooling layers, which provide translation invariance at the cost of discarding detailed positional information.
Capsule Networks (CapsNets) were designed to address this by replacing scalar neurons with vectors. These vectors represent the "pose" of an object part, allowing the network to naturally track transformations—a property known as equivariance.
While most CapsNet research has focused on supervised learning, the CoCa model explores their potential in unsupervised learning, where the goal is to learn representations from raw pixels by maximizing agreement between differently augmented views of the same image.
The CoCa Framework
The Contrastive Capsule (CoCa) model utilizes a Siamese network architecture consisting of two identical branches. Each branch processes a different augmented version of the same input image to learn invariant and equivariant features.
Architecture Components
- Encoder: A shallow 6-layer convolutional block (ConvBlock) acts as the initial feature extractor.
- PrimaryCaps: This layer converts the convolutional features into 16-dimensional vectors (capsules).
- ClassCaps: Using a "routing by agreement" mechanism, the model groups lower-level features into higher-level representations. This dynamic routing provides a form of top-down feedback, helping the model determine part-whole relationships during the forward pass.
- Contrastive Loss (NT-Xent): The model is trained to maximize the similarity between representations of the same image (positives) while minimizing similarity with other images in the batch (negatives).
Experimental Results
The model was evaluated on the CIFAR-10 dataset, consisting of 60,000 images across 10 classes. To evaluate the learned representations, a k-Nearest Neighbor (k-NN) approach was used, comparing test image features against a memory bank of training features.
Performance and Efficiency
- Accuracy: CoCa achieved a 70.50% top-1 accuracy and 98.10% top-5 accuracy.
- Parameter Efficiency: The model utilizes only 780,000 parameters, a 31x reduction compared to the 24.62 million parameters in SimCLR (ResNet-50).
- Computational Speed: CoCa requires 71x fewer FLOPs and trains 4x faster on identical hardware compared to current state-of-the-art unsupervised CNNs.
These results suggest that the inherent equivariance of capsules acts as a "shortcut," allowing the model to learn spatial properties that CNNs must learn through massive data augmentation and deeper architectures.
Conclusion and Future Work
CoCa demonstrates that Capsule Networks are highly effective for unsupervised representation learning, offering massive gains in computational efficiency. Future research will focus on addressing routing bottlenecks, such as the computational expense of dynamic routing, and scaling the model with larger batch sizes and advanced optimization techniques like LARS or Gaussian blur. The authors suggest that CoCa's spatial precision makes it particularly suitable for domains where labeled data is scarce but geometric accuracy is critical, such as medical imaging.
Summary updated