Introduction
Convolutional Neural Networks (convnets) trained on massive datasets have dramatically advanced the state-of-the-art in image classification and object detection. However, true visual understanding often requires establishing correspondence on a much finer level than simply identifying an object category. Because convnets utilize successive layers of pooling and possess large, overlapping "receptive fields," many researchers assumed that precise spatial information was averaged away or lost. The prevailing wisdom suggested that while convnets excel at coarse, image-level tasks, hand-engineered features like SIFT were still necessary for finding exact correspondences—such as matching the specific tip of a cat’s ear across two different photos.
This paper challenges that assumption, demonstrating that convnets trained solely for image classification implicitly learn to identify fine-grained spatial correspondences between object parts. Despite their architecture, they provide better localization and part-matching capabilities than traditional hand-engineered descriptors.
Methodology
The authors investigate convnet features (specifically from an AlexNet-style architecture pre-trained on ImageNet) through three primary approaches:
1. Feature Visualization: To understand what specific features represent, the authors employ a non-parametric reconstruction technique. For a given image patch, they find its nearest neighbors in "feature space" from a large database and average them.
2. Intraclass Alignment (Convnet Flow): The researchers adapt the "SIFT Flow" algorithm—which matches dense descriptors between images while enforcing a smoothness constraint—by replacing SIFT descriptors with convnet activations. They use a Markov Random Field (MRF) to solve for the best displacement field to align two different objects of the same category.
3. Keypoint Prediction:
Finally, they train sliding-window part detectors (linear SVMs) on conv5 features. To improve accuracy, they combine these detector scores with a "spatial prior"—a Gaussian distribution based on where that keypoint appeared in the most similar images from the training set.
Key Insights
- The "Center" Matters: Even though a
conv5neuron has a massive theoretical receptive field (e.g., 195x195 pixels), it remains most sensitive to the exact center of that field. Visualizations confirm that features localize at a much finer scale than their receptive field size implies. - Semantic vs. Visual Specificity: Moving deeper into the network (from
conv3toconv5), features transition from capturing "exact textures" to "semantic parts." Aconv3feature might match a specific pattern of fur, whereas aconv5feature matches "an eye," regardless of the animal's specific color, lighting, or texture. - Correspondence as a Byproduct: Strikingly, the network learns these precise spatial relationships implicitly. It was never explicitly trained on where an "eye" or a "wheel" was located; it learned to recognize these parts because they were discriminative for the higher-level task of whole-image classification.
Experimental Results
The authors evaluated their methods primarily using the PASCAL VOC 2011 dataset, leveraging keypoint annotations from the Berkeley Poselets dataset.
- Alignment: In the "Convnet Flow" experiments,
conv4features performed competitively with, or better than, SIFT. On the Percentage of Correct Keypoints (PCK) metric at a threshold of $\alpha=0.1$, convnet flow achieved 24.9%, edging out SIFT flow’s 24.7%. - Keypoint Classification: When classifying specific feature vectors (e.g., determining "is this a nose?"), convnet features consistently outperformed SIFT across almost all 20 PASCAL categories.
- Keypoint Prediction: Using
conv5features combined with a spatial prior yielded a mean accuracy of 42.5% PCK. This represents a massive leap over the SIFT-based baseline of 28.4%, demonstrating that convnet features are significantly more robust to the variations in pose and appearance found in real-world datasets.
Limitations and Future Work
While convnet features show remarkable implicit correspondence, the authors noted that localization precision is inherently limited by the network's "stride" (e.g., the 16-pixel jump between feature applications in deeper layers). They suggest that utilizing a finer stride or a regression-based approach could yield even greater precision. Furthermore, because these powerful features are learned from "weak" image-level labels, explicitly training for correspondence or using higher-resolution inputs could unlock even better performance in the future. Finally, at the time of publication, calculating dense "flow" with high-dimensional convnet features remained computationally expensive compared to traditional methods.
Summary updated