TL;DR
Most self-supervised learning (SSL) models are designed to be "invariant," meaning they learn to ignore transformations like rotation or scaling. This paper introduces a novel auxiliary task—intermediate transformation reconstruction—that forces models to retain this spatial information, creating "equivariant" features that significantly improve performance on dense prediction tasks like object detection and depth estimation without sacrificing classification accuracy.
What Problem Does This Paper Address?
In computer vision, there is a fundamental tension between invariance and equivariance.
- Invariance is great for classification: a "cat" is still a "cat" whether it is rotated 30 degrees or shifted to the left.
- Equivariance is essential for dense tasks: if an image rotates, the features representing it should "rotate" in a predictable way so the model can still pinpoint exactly where the object's boundaries or keypoints are.
Current state-of-the-art SSL methods (like DINOv2 or iBOT) are heavily biased toward invariance. While some recent "Split Invariant-Equivariant" (SIE) frameworks attempt to fix this, they often rely on rigid mathematical assumptions—like assuming transformations in the latent space must be linear. This paper tackles the challenge of learning flexible, "equivariance-coherent" representations that can handle complex, non-linear transformations without requiring restrictive architectural constraints.
Methods
The authors propose a framework that can be "plugged into" existing SSL methods. The core components are:
- Feature Splitting: The model's output feature vector is split into two parts: an invariant part ($z_{inv}$) and an equivariant part ($z_{equi}$).
- Intermediate Reconstruction Task: This is the "secret sauce." Instead of just matching two different views of an image, the model is tasked with reconstructing the image at intermediate steps along a transformation path. For example, if the data augmentation rotates an image by 30°, the model uses $z_{equi}$ to reconstruct what that image looked like at 10° and 20°.
- Lightweight Decoders: To keep the focus on the encoder's representation, the authors use very simple decoders (one linear layer followed by four convolutional layers) to perform the reconstruction.
- Hybrid Loss Function: The total loss combines a standard SSL loss (like iBOT's masked image modeling) applied to the invariant features and an $L_2$ reconstruction loss applied to the equivariant features.
By forcing the model to "see" the intermediate steps of a transformation, the equivariant features are naturally incentivized to store the geometric and photometric information that standard SSL usually discards.
Datasets
The models were pretrained on ImageNet-1K. Evaluation was performed across a massive suite of benchmarks:
- Synthetic Equivariance Benchmarks: A set of tasks designed to measure how well features track rotation, color jittering, blurring, and translation.
- Natural Image Classification: CIFAR-10/100, Oxford Pets, Food-101, Flowers-102, iNaturalist 2018, and ImageNet-1K (via linear probing).
- Dense Prediction Tasks: ADE20k (semantic segmentation), COCO (object detection/instance segmentation), MPII (keypoint detection), S-COCO (homography estimation), and NYU/KITTI (monocular depth estimation).
- Video Tasks: DAVIS 2017 and VIP (video object segmentation and part propagation).
Key Insights and Ideas
- The "Journey" Matters More than the Destination: By reconstructing intermediate points (e.g., 10° and 20°) rather than just the final augmented view (30°), the model treats these points as "anchors" that shape the geometry of the feature space.
- Removing the Linearity Constraint: Unlike previous methods that forced equivariant mappings to be linear, this reconstruction-based approach allows the model to learn complex, non-linear relationships, making it more robust to real-world distortions.
- Synergy, Not Sacrifice: Surprisingly, adding equivariant capabilities often improved classification performance on several datasets. This suggests that keeping some spatial/transformation context helps the model understand the "content" of the image better.
- Efficiency: The auxiliary task only adds about 7% to 15% computational overhead during training and zero overhead during inference (since the decoders are discarded).
Key Results
- Synthetic Superiority: On rotation prediction tasks, the proposed method improved the $R^2$ score of iBOT from 0.238 to 0.937, demonstrating a massive leap in capturing transformation info.
- State-of-the-Art Dense Prediction: When integrated with DINOv2, the method achieved 54.12 mIoU on ADE20k (segmentation), outperforming the vanilla DINOv2 baseline (53.49).
- Depth Estimation: On the KITTI dataset, the method significantly reduced the Absolute Relative Error (AbsRel) compared to standard iBOT and DINOv2.
- Video Consistency: The method showed strong gains in video object segmentation (DAVIS 2017), where maintaining feature coherence across frames is critical.
Limitations and Future Work
- Hyperparameter Tuning: The performance is sensitive to the size of the equivariant feature split ($d_{equi}$) and the weight of the reconstruction loss ($\lambda$). Finding the "sweet spot" requires some empirical testing.
- Architecture Focus: While the method is general, the paper focuses primarily on Vision Transformers (ViTs). Exploring how this behaves with newer architectures like Mamba or pure CNNs could be a future direction.
- Complex Deformations: While the method handles standard augmentations well, future work could explore even more complex, non-rigid transformations (like thin-plate splines or fluid-like warps) to further push the boundaries of equivariant learning.
Summary updated