Abstract
The equivariant behaviour of features is essential in many computer vision tasks, yet popular self-supervised learning (SSL) methods tend to constrain equivariance by design. We propose a self-supervised learning approach where the system learns transformations independently by reconstructing images that have undergone previously unseen transformations. Specifically, the model is tasked to reconstruct intermediate transformed images, e.g. translated or rotated images, without prior knowledge of these transformations. This auxiliary task encourages the model to develop equivariance-coherent features without relying on predefined transformation rules. To this end, we apply transformations to the input image, generating an image pair, and then split the extracted features into two sets per image. One set is used with a usual SSL loss encouraging invariance, the other with our loss based on the auxiliary task to reconstruct the intermediate transformed images. Our loss and the SSL loss are linearly combined with weighted terms. Evaluating on synthetic tasks with natural images, our proposed method strongly outperforms all competitors, regardless of whether they are designed to learn equivariance. Furthermore, when trained alongside augmentation-based methods as the invariance tasks, such as iBOT or DINOv2, we successfully learn a balanced combination of invariant and equivariant features. Our approach performs strong on a rich set of realistic computer vision downstream tasks, almost always improving over all baselines.
Introduction
In computer vision, a fundamental tension exists between invariance and equivariance. Invariance is crucial for classification tasks, where a "cat" remains a "cat" regardless of its orientation or position. Conversely, equivariance is essential for dense prediction tasks like object detection, semantic segmentation, and depth estimation, where features must track spatial transformations predictably to maintain spatial accuracy.
Current state-of-the-art self-supervised learning (SSL) methods, such as DINOv2 or iBOT, are heavily biased toward invariance by design. While some "Split Invariant-Equivariant" (SIE) frameworks have attempted to address this, they often rely on rigid mathematical assumptions, such as linear transformations in the latent space. This paper introduces a novel auxiliary task—intermediate transformation reconstruction—to learn flexible, equivariance-coherent representations that can handle complex, non-linear transformations without requiring restrictive architectural constraints.
Methodology
The proposed framework is designed to be integrated into existing SSL methods as a "plug-and-play" module. The architecture employs a feature splitting strategy where the encoder's output feature vector is divided into two parts: an invariant part ($z_{inv}$) and an equivariant part ($z_{equi}$).
The core innovation is the intermediate reconstruction task. Instead of simply matching two augmented views of an image, the model is tasked with reconstructing the image at intermediate steps along a transformation path. For example, if a data augmentation rotates an image by an angle $\phi$, the model uses $z_{equi}$ to reconstruct what that image looked like at intermediate angles like $\phi/3$ and $2\phi/3$.
This process utilizes lightweight decoders—typically a linear layer followed by a few convolutional layers—to perform the reconstruction. The total loss is a hybrid combination of a standard SSL loss (like iBOT's masked image modeling) applied to the invariant features and an $L_2$ reconstruction loss applied to the equivariant features. By forcing the model to "see" the intermediate steps of a transformation, the equivariant features are naturally incentivized to store the geometric and photometric information that standard SSL usually discards.
Experimental Setup
The models were pretrained on ImageNet-1K. Evaluation was conducted across a comprehensive suite of benchmarks to assess both invariant and equivariant properties:
- Synthetic Equivariance Benchmarks: Tasks designed to measure how well features track rotation, color jittering, blurring, and translation.
- Natural Image Classification: Linear probing on datasets like CIFAR-10/100, Oxford Pets, and ImageNet-1K.
- Dense Prediction Tasks: ADE20k (semantic segmentation), COCO (object detection), and NYU/KITTI (monocular depth estimation).
- Video Tasks: DAVIS 2017 for video object segmentation and part propagation.
Results and Discussion
The method demonstrates significant improvements in capturing transformation information. On synthetic rotation tasks, integrating this approach with iBOT increased the $R^2$ score from 0.238 to 0.937, representing a massive leap in capturing geometric information. For dense prediction, the method achieved 54.12 mIoU on ADE20k when combined with DINOv2, outperforming the vanilla DINOv2 baseline of 53.49.
A key insight from the results is the "Synergy, Not Sacrifice" effect: adding equivariant capabilities often improved classification performance on several datasets. This suggests that keeping some spatial and transformation context helps the model better understand the underlying content of the image. Furthermore, the auxiliary task is computationally efficient, adding only 7% to 15% overhead during training and zero overhead during inference, as the decoders are discarded after training.
Conclusion
By reconstructing the "journey" of a transformation rather than just the final augmented state, this method enables models to learn complex, non-linear equivariant features. While performance can be sensitive to hyperparameters like the feature split size ($d_{equi}$) and the reconstruction loss weight ($\lambda$), the framework offers a robust way to enhance SSL models for a wide range of downstream tasks. Future research may explore more complex deformations and the application of this method to emerging architectures beyond Vision Transformers.
Summary updated