Abstract
This work explores the use of spatial context as a source of free and plentiful supervisory signal for training a rich visual representation. Given only a large, unlabeled image collection, we extract random pairs of patches from each image and train a convolutional neural net to predict the position of the second patch relative to the first. We argue that doing well on this task requires the model to learn to recognize objects and their parts. We demonstrate that the feature representation learned using this within-image context captures visual similarity across images. Furthermore, we show that the learned ConvNet can be used in the R-CNN framework and provides a significant boost over a randomly-initialized ConvNet, resulting in state-of-the-art performance among algorithms which use only Pascal-provided training set annotations.
Introduction
Deep learning's success in computer vision has historically been tethered to massive, human-labeled datasets like ImageNet. However, labeling millions of images is expensive, slow, and doesn't scale to the billions of images available on the internet. The core challenge in moving toward unsupervised learning is defining a "pretext task": a proxy objective that forces the model to learn about the structure of the visual world, such as objects and boundaries, without explicit human-provided labels.
Methodology: Spatial Context as Supervision
Taking inspiration from the text domain (e.g., Word2Vec), where models learn word meanings by predicting a word's context, this approach translates the concept to images using spatial context.
The algorithm samples a "central" patch from an image and then samples a second patch from one of the eight possible neighboring locations in a 3x3 grid. Given these two patches, the model must classify which of the eight positions the second patch occupies relative to the first. The architecture uses a Siamese-style Convolutional Neural Network based on AlexNet, where each patch is processed through identical, weight-shared layers.
To prevent the model from "cheating" via trivial shortcuts like matching edges or textures across gaps, the authors introduced random gaps between patches and jittered their locations. They also discovered that the model could exploit chromatic aberration (tiny color shifts caused by camera lenses). To mitigate this, they implemented "color dropping" to remove these low-level signals, forcing the model to focus on high-level semantic features.
Evaluation and Results
The model was trained on the ImageNet (ILSVRC 2012) training set with all labels discarded. Evaluation was performed across several benchmarks:
- Object Detection: When fine-tuned for object detection on PASCAL VOC 2007, the self-supervised model achieved 46.3% mAP. This represented a ~5% boost over a model trained from scratch and outperformed contemporary unsupervised methods.
- Visual Discovery: Qualitatively, the model's features were able to cluster semantically similar objects—such as bird heads, car wheels, or faces—across different images, despite never being exposed to object labels.
- Scene Layout: On the Street View (Paris) dataset, the model demonstrated an ability to capture architectural elements and scene layout.
Discussion
The fundamental insight of this work is that spatial arrangement serves as a proxy for objecthood. To determine that a "cat ear" belongs above a "cat eye," the model must first learn to recognize these constituent parts. However, the model is a "lazy" learner; the discovery of the chromatic aberration shortcut highlights how neural networks will exploit any statistical fluke to minimize loss rather than learning high-level concepts.
While a performance gap remains between self-supervised and fully supervised approaches (54.2% mAP for ImageNet-pre-trained models), this work proves that unsupervised pre-training provides a massive head start. Future work involves scaling to even larger, noisier datasets where human labeling is impossible.
Summary updated