PUMP: Pyramidal and Uniqueness Matching Priors for Unsupervised Learning of Local Descriptors

Jérôme Revaud, Vincent Leroy, Philippe Weinzaepfel, Boris Chidlovskii

Introduction

Local image descriptors are essential components in computer vision, serving as the "digital fingerprints" for image patches. They enable critical tasks such as 3D reconstruction (Structure-from-Motion), SLAM, and visual localization by allowing algorithms to match points across different viewpoints. Traditionally, training these descriptors has required large-scale datasets with precise per-pixel correspondences. However, obtaining this ground truth often requires a functional SfM pipeline, creating a "chicken-and-egg" problem: you need good descriptors for SfM, but you need SfM to generate data for better descriptors.

PUMP (Pyramidal and Uniqueness Matching Priors) breaks this cycle by learning directly from unsupervised image pairs—images known to depict the same scene but lacking pre-defined pixel-to-pixel mappings.

Methodology

The PUMP framework replaces expensive ground truth labels with two fundamental geometric priors embedded directly into the loss function: Local Neighborhood Consistency and Uniqueness.

Local Neighborhood Consistency

This prior assumes that if two neighboring pixels in one image match two pixels in another, those matches should also be neighbors. PUMP enforces this through a non-parametric pyramidal aggregation strategy. It constructs a 4D correlation volume and iteratively aggregates matches across scales. This allows high-level "parent" patches to provide spatial context and guide the matching of low-level "child" pixels, ensuring geometric coherence.

Uniqueness Matching Loss ($L_U$)

In natural scenes, a point in one image typically corresponds to exactly one point in another. PUMP introduces a Uniqueness Matching Loss that normalizes the correlation volume and penalizes the model if a patch matches multiple locations. This encourages the network to produce highly discriminative features that avoid ambiguous correspondences.

The PUMP architecture: Features $F_1, F_2$ are extracted and used to build an initial correlation volume $C^0$, followed by iterative pyramidal construction and a unique matching loss.

These unsupervised priors are combined with a standard self-supervised loss (trained on synthetic homographies) to create a robust training signal capable of handling real-world image pairs without manual annotation.

Experimental Results

The model was trained on the SfM-120k dataset using only image pairs, ignoring the existing SfM-generated pixel correspondences for its primary loss. Evaluation was conducted across several challenging benchmarks:

  • Dense Matching (HPatches): PUMP achieved an Average End Point Error (AEPE) of 3.76, dramatically outperforming previous unsupervised state-of-the-art methods like WarpC (AEPE 21.00) and even surpassing supervised models like COTR (AEPE 7.98).
  • Visual Localization (Aachen Day-Night): The model demonstrated exceptional robustness to extreme illumination changes. When paired with a SIFT keypoint detector, PUMP improved localization accuracy at the 0.5m threshold by 10% absolute compared to models trained only on synthetic data.
Qualitative comparison showing the impact of the unsupervised loss. The PUMP framework significantly increases the number of true matches while reducing false positives.

Discussion and Conclusion

PUMP demonstrates the "unreasonable effectiveness" of geometric priors. By hard-coding the structure of the matching process while keeping the descriptors learnable, the framework bridges the gap between synthetic training and real-world complexity. The results suggest that unsupervised models can outperform fully-supervised counterparts by leveraging the vast potential of unlabeled image pairs, particularly in scenarios where traditional SfM pipelines fail.

While PUMP significantly reduces the need for labels, it still requires image pairs of the same scene and may face challenges in perfectly untextured or highly repetitive environments. Future research may focus on integrating global context to resolve these local ambiguities.

Summary updated

Abstract

Existing approaches for learning local image descriptors have shown remarkable achievements in a wide range of geometric tasks. However, most of them require per-pixel correspondence-level supervision, which is difficult to acquire at scale and in high quality. In this paper, we propose to explicitly integrate two matching priors in a single loss in order to learn local descriptors without supervision. Given two images depicting the same scene, we extract pixel descriptors and build a correlation volume. The first prior enforces the local consistency of matches in this volume via a pyramidal structure iteratively constructed using a nonparametric module. The second prior exploits the fact that each descriptor should match with at most one descriptor from the other image. We combine our unsupervised loss with a standard self-supervised loss trained from synthetic image augmentations. Feature descriptors learned by the proposed approach outperform their fully- and selfsupervised counterparts on various geometric benchmarks such as visual localization and image matching, achieving state-of-the-art performance. Project webpage: https: //europe.naverlabs.com/research/3d-vision/pump.
BibTeX
@inproceedings{revaud2022pump,
  author = {Jérôme Revaud and Vincent Leroy and Philippe Weinzaepfel and Boris Chidlovskii},
  title = {{PUMP: Pyramidal and Uniqueness Matching Priors for Unsupervised Learning of Local Descriptors}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00390},
}
Figure