Do Convnets Learn Correspondence?

Jonathan Long, Ning Zhang, Trevor Darrell

Introduction

Convolutional Neural Networks (convnets) trained on massive datasets have dramatically advanced the state-of-the-art in image classification and object detection. However, true visual understanding often requires establishing correspondence on a much finer level than simply identifying an object category. Because convnets utilize successive layers of pooling and possess large, overlapping "receptive fields," many researchers assumed that precise spatial information was averaged away or lost. The prevailing wisdom suggested that while convnets excel at coarse, image-level tasks, hand-engineered features like SIFT were still necessary for finding exact correspondences—such as matching the specific tip of a cat’s ear across two different photos.

This paper challenges that assumption, demonstrating that convnets trained solely for image classification implicitly learn to identify fine-grained spatial correspondences between object parts. Despite their architecture, they provide better localization and part-matching capabilities than traditional hand-engineered descriptors.

Methodology

The authors investigate convnet features (specifically from an AlexNet-style architecture pre-trained on ImageNet) through three primary approaches:

1. Feature Visualization: To understand what specific features represent, the authors employ a non-parametric reconstruction technique. For a given image patch, they find its nearest neighbors in "feature space" from a large database and average them.

Feature visualization across layers

2. Intraclass Alignment (Convnet Flow): The researchers adapt the "SIFT Flow" algorithm—which matches dense descriptors between images while enforcing a smoothness constraint—by replacing SIFT descriptors with convnet activations. They use a Markov Random Field (MRF) to solve for the best displacement field to align two different objects of the same category.

Convnet Flow vs SIFT Flow

3. Keypoint Prediction: Finally, they train sliding-window part detectors (linear SVMs) on conv5 features. To improve accuracy, they combine these detector scores with a "spatial prior"—a Gaussian distribution based on where that keypoint appeared in the most similar images from the training set.

Key Insights

  • The "Center" Matters: Even though a conv5 neuron has a massive theoretical receptive field (e.g., 195x195 pixels), it remains most sensitive to the exact center of that field. Visualizations confirm that features localize at a much finer scale than their receptive field size implies.
  • Semantic vs. Visual Specificity: Moving deeper into the network (from conv3 to conv5), features transition from capturing "exact textures" to "semantic parts." A conv3 feature might match a specific pattern of fur, whereas a conv5 feature matches "an eye," regardless of the animal's specific color, lighting, or texture.
  • Correspondence as a Byproduct: Strikingly, the network learns these precise spatial relationships implicitly. It was never explicitly trained on where an "eye" or a "wheel" was located; it learned to recognize these parts because they were discriminative for the higher-level task of whole-image classification.

Experimental Results

The authors evaluated their methods primarily using the PASCAL VOC 2011 dataset, leveraging keypoint annotations from the Berkeley Poselets dataset.

  • Alignment: In the "Convnet Flow" experiments, conv4 features performed competitively with, or better than, SIFT. On the Percentage of Correct Keypoints (PCK) metric at a threshold of $\alpha=0.1$, convnet flow achieved 24.9%, edging out SIFT flow’s 24.7%.
  • Keypoint Classification: When classifying specific feature vectors (e.g., determining "is this a nose?"), convnet features consistently outperformed SIFT across almost all 20 PASCAL categories.
  • Keypoint Prediction: Using conv5 features combined with a spatial prior yielded a mean accuracy of 42.5% PCK. This represents a massive leap over the SIFT-based baseline of 28.4%, demonstrating that convnet features are significantly more robust to the variations in pose and appearance found in real-world datasets.

Limitations and Future Work

While convnet features show remarkable implicit correspondence, the authors noted that localization precision is inherently limited by the network's "stride" (e.g., the 16-pixel jump between feature applications in deeper layers). They suggest that utilizing a finer stride or a regression-based approach could yield even greater precision. Furthermore, because these powerful features are learned from "weak" image-level labels, explicitly training for correspondence or using higher-resolution inputs could unlock even better performance in the future. Finally, at the time of publication, calculating dense "flow" with high-dimensional convnet features remained computationally expensive compared to traditional methods.

Summary updated

Abstract

Convolutional neural nets (convnets) trained from massive labeled datasets have substantially improved the state-of-the-art in image classification and object detection. However, visual understanding requires establishing correspondence on a finer level than object category. Given their large pooling regions and training from whole-image labels, it is not clear that convnets derive their success from an accurate correspondence model which could be used for precise localization. In this paper, we study the effectiveness of convnet activation features for tasks requiring correspondence. We present evidence that convnet features localize at a much finer scale than their receptive field sizes, that they can be used to perform intraclass aligment as well as conventional hand-engineered features, and that they outperform conventional features in keypoint prediction on objects from PASCAL VOC 2011.
BibTeX
@inproceedings{long2014do,
  author = {Jonathan Long and Ning Zhang and Trevor Darrell},
  title = {{Do Convnets Learn Correspondence?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
  url = {https://openreview.net/forum?id=BJbm6vZuZH},
}
Figure