Abstract
This paper proposes a method to construct pretext tasks for self-supervised learning (SSL) on group equivariant neural networks (G-CNNs). G-CNNs are models whose architectural structure is restricted to commute with transformations on the input. To ensure that training is consistent with this equivariance, the authors propose two concepts: equivariant pretext labels and invariant contrastive loss. Equivariant pretext labels use a set of labels that transform in correspondence with input changes, while invariant contrastive loss utilizes a modified similarity metric that absorbs the effect of transformations. Experiments on standard image recognition benchmarks demonstrate that G-CNNs significantly benefit from these tailored self-supervised tasks.
Introduction
Self-supervised learning has become a cornerstone of modern computer vision, enabling feature extraction from vast unlabeled datasets. However, standard SSL pretext tasks are typically designed for non-equivariant architectures like vanilla ResNets. Group Equivariant Neural Networks (G-CNNs) are specialized models designed to preserve geometric symmetries—ensuring that if an input rotates, its internal representation transforms predictably.
A fundamental conflict arises when standard SSL is applied to G-CNNs: many pretext tasks (e.g., predicting the absolute orientation of a patch) require the network to learn mappings that violate the very equivariance the G-CNN is built to preserve. This paper resolves this friction by making the SSL objectives "speak the same language" as the equivariant models.
Methodology
The authors introduce two primary strategies to align SSL with the inductive biases of G-CNNs.
Equivariant Pretext Labels
In tasks such as Context Prediction or Jigsaw Puzzles, the target labels must change predictably when the input is transformed. For an equivariant model $f_{\theta}$ to learn a pretext task $y(x)$, the task must satisfy the consistency condition: $y(T_{in}(g)x) = T_{out}(g)y(x)$. This ensures the training signal does not contradict the model's internal symmetry constraints.
Invariant Contrastive Loss
In contrastive learning frameworks like MoCo or SimSiam, the goal is to maximize similarity between different views of the same image. To make this compatible with G-CNNs, the authors modify the similarity metric to be invariant under the group action $G$. Instead of a standard dot product, they calculate the average inner product over all possible transformations in the group:
$$\text{Sim}(x, k) = \frac{1}{|G|} \sum_{g \in G} (T_{out}(g)f_{\theta}(x))^t (k)$$
This ensures the loss remains constant regardless of the specific orientation of the input features, allowing the G-CNN to converge on a more robust representation.
Experimental Results
The proposed methods were evaluated on ImageNet-1k pretraining with downstream tasks on VOC2007 and iNaturalist18. The equivariant approach consistently outperformed standard baselines:
- ImageNet (Linear Probe): The Jigsaw task saw a significant jump from 35.1% (baseline) to 43.1%. Contrastive methods like SimSiam and MoCo v2 improved by approximately 2%.
- VOC2007: Equivariant models showed superior transferability, particularly in patch-based tasks, outperforming standard ResNet baselines by 2-4%.
- Ablation Studies: The authors found that simply using a G-CNN with a standard SSL loss often performs worse than a standard CNN. The performance boost is only realized when the loss function is also made equivariant or invariant.
Conclusion
This work demonstrates that the effectiveness of G-CNNs in self-supervised settings depends on the alignment between architectural symmetries and the training objective. By introducing equivariant pretext labels and invariant contrastive losses, the authors provide a framework to unlock the full potential of symmetry-aware architectures in large-scale feature learning. Future work will focus on reducing the computational overhead of G-CNNs and extending these principles to continuous symmetries.
Summary updated