Generalized Contrastive Optimization of Siamese Networks for Place\n Recognition

Luz María, Nicola Strisciuglio, Nicolai Petkov

Abstract

Visual place recognition is a challenging task in computer vision and a key component of camera-based localization and navigation systems. Recently, Convolutional Neural Networks (CNNs) achieved high results and good generalization capabilities. They are usually trained using pairs or triplets of images labeled as either similar or dissimilar, in a binary fashion. In practice, the similarity between two images is not binary, but continuous. Furthermore, training these CNNs is computationally complex and involves costly pair and triplet mining strategies. We propose a Generalized Contrastive loss (GCL) function that relies on image similarity as a continuous measure, and use it to train a siamese CNN. Furthermore, we present three techniques for automatic annotation of image pairs with labels indicating their degree of similarity, and deploy them to re-annotate the MSLS, TB-Places, and 7Scenes datasets. We demonstrate that siamese CNNs trained using the GCL function and the improved annotations consistently outperform their binary counterparts. Our models trained on MSLS outperform the state-of-the-art methods, including NetVLAD, NetVLAD-SARE, AP-GeM and Patch-NetVLAD, and generalize well on the Pittsburgh30k, Tokyo 24/7, RobotCar Seasons v2 and Extended CMU Seasons datasets. Furthermore, training a siamese network using the GCL function does not require complex pair mining.

Moving Beyond Binary Labels

Visual Place Recognition (VPR) is fundamental for localization: given a query image, a system must identify its location by finding matches in a database. Traditionally, deep learning models for VPR are trained using binary labels, where image pairs are strictly categorized as "similar" or "dissimilar."

However, real-world scenes exhibit continuous degrees of overlap. Two images might share 80% of a scene from slightly different viewpoints or share 0% overlap while being at the same intersection facing opposite directions. Forcing these nuanced relationships into a binary classification creates inconsistent training signals. To mitigate this, researchers often employ "hard mining"—complex, memory-intensive algorithms that search for the most challenging image pairs. This paper introduces a more elegant solution: treating similarity as a continuous spectrum.

Generalized Contrastive Loss and Continuous Labeling

The authors propose the Generalized Contrastive Loss (GCL), which replaces the binary label $y$ with a continuous similarity measure $\psi \in [0, 1]$. If two images have a 50% overlap, GCL ensures their embeddings are placed moderately close, providing a smoother and more informative gradient than binary loss functions.

Varying degrees of image similarity between a query (a) and potential matches (b, c, d).

To support this loss function, three automatic labeling techniques were developed to quantify similarity ($\psi$):

  • Weak 2D Overlap: Estimates intersection-over-union (IoU) using GPS and compass data.
  • Strong 2D Overlap: Utilizes precise 6DOF camera poses.
  • 3D Overlap: Calculates the percentage of shared 3D points from point cloud reconstructions.

By using GCL, the need for complex hard-pair mining is eliminated. A simple "naïve mining" strategy—ensuring a balanced distribution of similarity scores in each batch—is sufficient for effective training.

Overcoming the Transitivity Paradox

A significant advantage of GCL is its ability to resolve the "transitivity paradox." In binary training, if Image A matches B, and B matches C, the model is pressured to force A and C together in the embedding space. However, if A and C share no visual features (e.g., they are linked only by the intermediate view B), this forcing leads to "feature drift."

Conceptual illustration of the transitivity paradox where A and C share no features but are linked by B.

GCL acknowledges that A and C are dissimilar, preventing the collapse of distinct features. This leads to a more organized latent space where the distance between embeddings is proportional to actual geometric overlap.

Performance and Generalization

The researchers re-annotated standard benchmarks (MSLS, TB-Places, 7Scenes) and found that GCL-trained models consistently outperformed binary counterparts.

  • State-of-the-Art Results: On the MSLS test set, their ResNeXt-GeM-GCL model achieved a top-5 recall of 76.2%, significantly surpassing NetVLAD (44%) and other prominent methods like AP-GeM.
  • Robust Generalization: Models trained on MSLS generalized exceptionally well to unseen datasets like Pittsburgh30k and Tokyo 24/7, even across different seasons and lighting conditions.
  • Efficiency: The simplified training process is remarkably faster. Their model reached convergence in just 14 hours on a single GPU, compared to the 22 days required for NetVLAD-16.

Conclusion

By shifting from binary to continuous similarity labels, the Generalized Contrastive Loss provides a more accurate and efficient framework for training VPR models. While challenges remain—particularly in extreme nighttime conditions with limited training data—this approach significantly advances the state-of-the-art in camera-based localization.

Summary updated

Abstract

Visual place recognition is a challenging task in computer vision and a key\ncomponent of camera-based localization and navigation systems. Recently,\nConvolutional Neural Networks (CNNs) achieved high results and good\ngeneralization capabilities. They are usually trained using pairs or triplets\nof images labeled as either similar or dissimilar, in a binary fashion. In\npractice, the similarity between two images is not binary, but continuous.\nFurthermore, training these CNNs is computationally complex and involves costly\npair and triplet mining strategies. We propose a Generalized Contrastive loss\n(GCL) function that relies on image similarity as a continuous measure, and use\nit to train a siamese CNN. Furthermore, we present three techniques for\nautomatic annotation of image pairs with labels indicating their degree of\nsimilarity, and deploy them to re-annotate the MSLS, TB-Places, and 7Scenes\ndatasets. We demonstrate that siamese CNNs trained using the GCL function and\nthe improved annotations consistently outperform their binary counterparts. Our\nmodels trained on MSLS outperform the state-of-the-art methods, including\nNetVLAD, NetVLAD-SARE, AP-GeM and Patch-NetVLAD, and generalize well on the\nPittsburgh30k, Tokyo 24/7, RobotCar Seasons v2 and Extended CMU Seasons\ndatasets. Furthermore, training a siamese network using the GCL function does\nnot require complex pair mining. We release the source code at\nhttps://github.com/marialeyvallina/generalized_contrastive_loss.\n
BibTeX
@inproceedings{maria2023generalized,
  author = {Luz María and Nicola Strisciuglio and Nicolai Petkov},
  title = {{Generalized Contrastive Optimization of Siamese Networks for Place\textbackslash\{\}n Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=t8oqvcrUg9},
}
Figure