Abstract
Patch-level descriptors underlie several important computer vision tasks, such as stereo-matching or content-based image retrieval. We introduce a deep convolutional architecture that yields patch-level descriptors, as an alternative to the popular SIFT descriptor for image retrieval. The proposed family of descriptors, called Patch-CKN, adapt the recently introduced Convolutional Kernel Network (CKN), an unsupervised framework to learn convolutional architectures. We present a comparison framework to benchmark current deep convolutional approaches along with Patch-CKN for both patch and image retrieval, including our novel "RomePatches" dataset. Patch-CKN descriptors yield competitive results compared to supervised CNN alternatives on patch and image retrieval.
Introduction
In computer vision, image retrieval involves finding images in a database that contain the same object or scene as a query image. This process typically relies on three steps: detecting interest points, describing the surrounding patches, and matching these descriptors. For decades, hand-crafted algorithms like SIFT were the gold standard. While modern Convolutional Neural Networks (CNNs) have surpassed SIFT in many areas, they generally require massive amounts of labeled data, which is difficult to obtain for local image patches.
This paper introduces Patch-CKN, a deep convolutional architecture trained without human-labeled data. By leveraging an unsupervised kernel-based approach, Patch-CKN achieves performance competitive with supervised CNNs and traditional SIFT descriptors for both patch matching and large-scale image retrieval.
Methodology: Convolutional Kernel Networks
The core of this approach is the Convolutional Kernel Network (CKN). Unlike standard CNNs that minimize classification error via backpropagation, CKNs are rooted in kernel methods and learn filters by approximating a "match kernel"—a function measuring similarity between patches.
The architecture typically employs two layers. The first layer captures low-level features, while the second aggregates them for a more global view of the local patch. The authors explored three input variants:
- CKN-raw: Captures color using raw RGB values.
- CKN-white: Uses PCA-whitening to focus on local variations, enhancing robustness to global color shifts.
- CKN-grad: Operates on image gradients, effectively serving as a deep counterpart to SIFT.
To scale these local descriptors to full-image retrieval, the authors utilize VLAD (Vector of Locally Aggregated Descriptors) to aggregate patch-level information into a single global vector.
Experimental Setup and Datasets
The proposed descriptors were evaluated on several benchmarks:
- RomePatches: A new dataset introduced in this paper containing 20,000 patches from 16,000 Flickr images of Rome, with ground truth established via 3D reconstruction.
- Mikolajczyk Dataset: Evaluates robustness to rotation, blur, and compression.
- Oxford Buildings, UKbench, and Holidays: Standard benchmarks for large-scale image retrieval.
Key Results and Insights
The experiments yielded several critical insights:
- Unsupervised Depth: Hierarchical structures are valuable even without labels. Patch-CKN captures expressive power comparable to supervised networks.
- Optimal Complexity: For local patches, intermediate layers (similar to layer 4 of AlexNet) often outperform deeper layers. Patch-CKN is designed to operate in this "sweet spot."
- Efficiency: Training is remarkably fast; a CKN can be trained in approximately 10 minutes on a single GPU, compared to days for supervised alternatives.
In patch matching on RomePatches, CKN-grad achieved a mean Average Precision (mAP) of 88.1%, outperforming SIFT. For image retrieval on the Holidays dataset, a combination of variants (CKN-mix) reached 79.3% mAP, matching state-of-the-art supervised features.
Conclusion
Patch-CKN demonstrates that high-performing local features can be learned without supervision. While challenges remain regarding color sensitivity in certain architectural contexts and high descriptor dimensionality, the speed and performance of CKNs make them a compelling alternative to supervised CNNs. Future work may involve co-optimizing interest point detection with the description phase using similar unsupervised frameworks.
Summary updated