Abstract
This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context indeed captures visual similarity across images. For example, this representation allows us to perform unsupervised visual discovery of objects like cats, people, and even birds from the Pascal VOC 2011 detection dataset. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.
Introduction
Modern deep learning thrives on massive datasets like ImageNet, but these require millions of manually labeled examples—an expensive and unscalable process. Purely unsupervised learning, which tries to find patterns in data without labels, historically struggled to capture the high-level "meaning" of images, often getting stuck on low-level textures or pixel statistics.
The authors address this by looking at how Natural Language Processing (NLP) uses context. In text, you can learn word meanings by predicting a word's neighbors (e.g., Word2Vec). This paper asks: Can we use the spatial context of an image as a "free" supervisory signal? If a model can tell that an "ear" patch usually sits above and to the side of a "nose" patch, it must eventually learn what ears and noses look like.
The Context Prediction Task
The core of the paper is a relative position prediction task. For any given image, the algorithm samples a central "seed" patch and then samples a second patch from one of the eight surrounding grid positions. The model is shown these two patches (with no information about their original location) and must perform an 8-way classification to guess where the second patch was relative to the first (e.g., "top-right" or "bottom-center").
To solve this, the authors use a "late-fusion" Siamese-style ConvNet based on AlexNet. Two identical towers process the two patches separately up to the first fully connected layer, where features are concatenated for the final prediction. A major contribution is identifying how neural networks "cheat" at this task. To prevent the network from simply matching lines or using lens artifacts like chromatic aberration, the authors introduced spatial jitter between patches and "color-dropped" images during training.
Evaluation and Results
The model was trained on the ImageNet 2012 training set (approx. 1.3 million images) with all labels discarded. It was then evaluated on benchmarks like Pascal VOC 2007 for object detection and NYUv2 for surface normal estimation.
On Pascal VOC 2007, the self-supervised model achieved 46.3% mAP, a massive jump over a randomly initialized network (~40%). Beyond detection, the model successfully performed visual data mining, clustering semantically similar objects (like cats, birds, and wheels) across different images without ever being told what those objects were.
One of the most surprising findings was that this method performed nearly as well as supervised models on geometry-based tasks, suggesting that supervised classification might actually encourage models to ignore helpful geometric information.
Conclusion
This paper demonstrates that you don't need semantic labels to learn semantic features. The "pretext task" acts as a proxy that forces the model to understand object composition. While a performance gap remains between self-supervised and fully supervised pre-training, the discovery that a network could "smell" lens artifacts to solve a spatial puzzle highlights the importance of careful data augmentation. Future directions include more complex tasks like full jigsaw puzzles or combining context with other signals like video motion.
Summary updated