Abstract
Existing co-localization techniques significantly lose performance over weakly or fully supervised methods in accuracy and inference time. In this paper, we overcome common drawbacks of co-localization techniques by utilizing self-supervised learning approach. The major technical contributions of the proposed method are two-fold. 1) We devise a new geometric transformation, namely point symmetric transformation and utilize its parameters as an artificial label for self-supervised learning. This new transformation can also play the role of region-drop based regularization. 2) We suggest a heat map extraction method for computing the heat map from the network trained by self-supervision, namely class-agnostic activation mapping. It is done by computing the spatial attention map. Based on extensive evaluations, we observe that the proposed method records new state-of-the-art performance in three fine-grained datasets for unsupervised object localization. Moreover, we show that the idea of the proposed method can be adopted in a modified manner to solve the weakly supervised object localization task. As a result, we outperform the current state-of-the-art technique in weakly supervised object localization by a significant gap.
Introduction
Object localization—identifying exactly where an object is in an image—usually requires expensive pixel-level annotations like bounding boxes. To lower costs, researchers use Weakly Supervised Object Localization (WSOL), which only needs image-level labels. However, the most challenging frontier is unsupervised co-localization, where the algorithm is given a collection of images of the same class but has no labels at all.
Existing co-localization methods are often slow and inaccurate because they tend to focus only on the most discriminative part of an object—like a bird’s beak—rather than the whole body. PsyNet is a novel self-supervised framework that achieves state-of-the-art object localization without requiring any human-annotated labels. By training a network to predict a custom "Point Symmetric Transformation" (PST), the model learns to identify entire objects, outperforming even some weakly supervised methods while remaining fast enough for real-time applications.
Methods
The PsyNet framework introduces two primary innovations to solve the localization problem through self-supervision:
Point Symmetric Transformation (PST)
PST is a new geometric transformation used as a "pretext task." The image is subjected to a large translation (shifting it by half its resolution) and the resulting empty space is filled using reflection padding. PST acts as a form of "regional dropout." By shifting the image significantly, parts of the object are effectively "erased" or moved. To correctly predict the transformation parameters, the network is forced to look at the entire object, not just the most obvious parts. Reflection padding is crucial here as it maintains natural image statistics, preventing the network from "cheating" by identifying artificial edges.
Class-Agnostic Activation Mapping (CAAM)
Standard Class Activation Mapping (CAM) requires class labels to generate a heat map. Since PsyNet is unsupervised, it utilizes CAAM, which aggregates spatial attention from the feature maps of the self-supervised network. By performing channel pooling on the last convolutional layer, it identifies which regions the network used to guess the transformation, which naturally corresponds to the object of interest.
Key Results
PsyNet was evaluated on three fine-grained benchmarks: CUB200-2011 (birds), Stanford Cars, and FGVC Aircraft. These datasets are challenging because models often over-focus on small, class-specific details.
- Accuracy: On the CUB200 dataset, PsyNet achieved a GT-Known localization accuracy of 83.78% (using VGG16), significantly beating the previous state-of-the-art. With a stronger backbone (SE-ResNet50), it reached 85.10%.
- Speed: The inference time is approximately 0.0067 seconds per image on a GPU—roughly 30 times faster than competing methods like MO, making it viable for real-time use.
- WSOL Performance: When applied to a weakly supervised setting, PsyNet outperformed existing techniques by a significant gap, achieving nearly a 10% performance gain in some metrics.
Conclusion
PsyNet demonstrates that predicting geometric shifts is a powerful pretext task for learning object boundaries. By forcing the network to understand the entire structure of an object to solve the PST task, the model avoids the common pitfall of focusing only on discriminative parts. Future work will focus on multi-object scenarios and moving toward a truly "from scratch" self-supervised pipeline that does not rely on ImageNet pre-training.
Summary updated