Data-Efficient Large Scale Place Recognition With Graded Similarity Supervision

Maria Leyva-Vallina, Nicola Strisciuglio, Nicolai Petkov

Abstract

Visual place recognition (VPR) is a fundamental task of computer vision for visual localization. Existing methods are trained using image pairs that either depict the same place or not. Such a binary indication does not consider continuous relations of similarity between images of the same place taken from different positions, determined by the continuous nature of camera pose. The binary similarity induces a noisy supervision signal into the training of VPR methods, which stall in local minima and require expensive hard mining algorithms to guarantee convergence. Motivated by the fact that two images of the same place only partially share visual cues due to camera pose differences, we deploy an automatic re-annotation strategy to re-label VPR datasets. We compute graded similarity labels for image pairs based on available localization metadata. Furthermore, we propose a new Generalized Contrastive Loss (GCL) that uses graded similarity labels for training contrastive networks. We demonstrate that the use of the new labels and GCL allow to dispense from hard-pair mining, and to train image descriptors that perform better in VPR by nearest neighbor search, obtaining superior or comparable results than methods that require expensive hard-pair mining and re-ranking techniques.

Introduction: The Flaw in Binary Supervision

Visual Place Recognition (VPR) is the process of matching a query image to a specific location in a database—essentially the "Where am I?" of computer vision. Traditionally, deep learning models for VPR have relied on binary supervision: two images are labeled as a "positive" match if they are within a fixed distance (e.g., 25 meters) and a "negative" non-match otherwise.

Visualizing graded similarity: (a) is the query, while (b), (c), and (d) represent potential matches with varying degrees of geometric overlap and visual similarity.

However, this binary approach is fundamentally flawed. In real-world scenarios, two images might have 90% overlap or only 10% overlap, yet binary systems treat them identically. This "noisy" supervision often forces the use of complex "hard-pair mining" strategies—searching through thousands of images during training to find the most confusing examples—to ensure model convergence. These strategies are computationally expensive, memory-intensive, and significantly slow down the training process.

Methodology: Graded Similarity and GCL

The core contribution of this work is a shift from binary to graded similarity supervision. By acknowledging that similarity is a continuous spectrum based on camera overlap, the authors introduce two key innovations:

1. Generalized Contrastive Loss (GCL)

The authors modified the standard contrastive loss function to incorporate a continuous variable $\psi \in [0, 1]$, representing the degree of similarity. As shown in the architecture diagram below, the GCL optimizes the distance between image representations in the latent space proportionally to their actual geometric overlap.

The Generalized Contrastive Loss (GCL) framework uses a Siamese network to compute image representations, supervised by continuous similarity scores rather than binary labels.

2. Automatic Annotation and Naïve Mining

To generate these $\psi$ values without manual labor, the authors proposed three automatic techniques based on available metadata:

  • Weak 2D Field-of-View (FoV): Uses GPS and compass angles.
  • Strong 2D FoV: Uses precise 6DOF camera poses.
  • 3D FoV: Uses 3D point clouds to calculate the intersection-over-union of the camera frustums.

Because GCL provides a richer training signal, the authors were able to replace complex mining with "Naïve Mining," simply ensuring each batch contains a balanced mix of positive, "soft" negative (partial overlap), and "hard" negative pairs.

Experimental Setup

The authors re-annotated several standard datasets with graded similarity labels for evaluation:

  • MSLS (Mapillary Street Level Sequences): A massive outdoor dataset for primary training.
  • 7Scenes: An indoor benchmark for 3D-based similarity.
  • TB-Places: A dataset featuring garden environments with repetitive textures.
  • Generalization Sets: Models were tested on Pittsburgh30k, Tokyo 24/7, RobotCar Seasons v2, and Extended CMU Seasons to evaluate performance in unseen environments.

Key Results and Insights

  • State-of-the-Art Performance: On the MSLS test set, the ResNeXt-GeM-GCL model achieved a 76.2% Top-5 recall, significantly outperforming established methods like NetVLAD (44%) and Patch-NetVLAD (57.6%).
  • Efficiency and Speed: By removing the need for hard-pair mining, training complexity was drastically reduced. A ResNet50 model trained with GCL takes only 6 hours on MSLS, compared to the days required for traditional methods.
  • Intrinsic Regularization: The GCL gradient scales based on similarity. When images have low overlap, the model does not force their representations to be identical, preventing the latent space from becoming warped or overfitted to specific viewpoints.
  • Robust Feature Focus: Activation maps indicate that GCL-trained models focus more on permanent structures (like buildings) and ignore transient objects (like cars) more effectively than binary-trained counterparts.

Conclusion and Future Work

While GCL significantly improves VPR efficiency and accuracy, some limitations remain. The current similarity measure is strictly tied to camera geometry and does not yet account for semantic similarity (e.g., identical-looking storefronts in different locations). Future work includes integrating GCL into full end-to-end 6DOF localization pipelines and improving performance on underrepresented data, such as night-time scenes.

Summary updated

Abstract

Visual place recognition (VPR) is a fundamental task of computer vision for visual localization. Existing methods are trained using image pairs that either depict the same place or not. Such a binary indication does not consider continuous relations of similarity between images of the same place taken from different positions, determined by the continuous nature of camera pose. The binary similarity induces a noisy supervision signal into the training of VPR methods, which stall in local minima and require expensive hard mining algorithms to guarantee convergence. Motivated by the fact that two images of the same place only partially share visual cues due to camera pose differences, we deploy an automatic re-annotation strategy to re-label VPR datasets. We compute graded similarity labels for image pairs based on available localization metadata. Furthermore, we propose a new Generalized Contrastive Loss (GCL) that uses graded similarity labels for training contrastive networks. We demonstrate that the use of the new labels and GCL allow to dispense from hard-pair mining, and to train image descriptors that perform better in VPR by nearest neighbor search, obtaining superior or comparable results than methods that require expensive hard-pair mining and re-ranking techniques.
BibTeX
@inproceedings{leyvavallina2023data,
  author = {Maria Leyva-Vallina and Nicola Strisciuglio and Nicolai Petkov},
  title = {{Data-Efficient Large Scale Place Recognition With Graded Similarity Supervision}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=t8oqvcrUg9},
}
Figure