ETR: An Efficient Transformer for Re-ranking in Visual Place Recognition

Hao Zhang, Xin Chen, Heming Jing, Yingbin Zheng, Yuan Wu, Cheng Jin

Introduction

Visual Place Recognition (VPR) is a fundamental task in robotics and computer vision, enabling systems to determine their location by matching a query image against a vast database of geo-tagged images. To balance efficiency and accuracy, modern VPR systems typically employ a two-stage pipeline. First, a Global Retrieval stage uses compact global descriptors to identify a set of top-$k$ potential matches. Second, a Re-ranking stage performs a fine-grained comparison using local features to refine the results.

While effective, the re-ranking stage has traditionally been a significant computational bottleneck. Conventional methods rely on geometric verification (e.g., RANSAC) to align local features. These iterative processes are computationally expensive, difficult to parallelize, and often too slow for real-time applications like autonomous driving. This paper introduces the Efficient Transformer for Re-ranking (ETR), a novel architecture that replaces traditional geometric solvers with a fast, learnable Transformer-based module.

The ETR Architecture

The core innovation of ETR is the use of self- and cross-attention mechanisms to implicitly learn spatial correspondences between image pairs. Unlike RANSAC, which explicitly solves for a geometric transformation, ETR processes image pairs in a single forward pass, making it highly amenable to GPU acceleration and parallel processing.

The ETR architecture overview, illustrating the flow from input images through CNN feature extraction and segment encoding to the ETR blocks and final similarity score.

The model utilizes two types of attention within its ETR Blocks:

  • Self-Attention (SA): Captures complex spatial relationships between local features within each individual image.
  • Cross-Attention (CA): Enables bidirectional information exchange between the query and candidate images, allowing the model to identify matching regions and calculate a similarity score.

To distinguish between global and local features, the authors incorporate Segment Encoding, similar to the approach used in BERT. The model is versatile and can be implemented using different feature extractors, such as DELG (ETR-D) or SuperPoint (ETR-S).

Experimental Results

The ETR framework was evaluated on several challenging VPR benchmarks, including MSLS (Mapillary Street-level Sequences), Pitts30k, and Tokyo 24/7. The results demonstrate that ETR significantly outperforms existing re-ranking methods in both speed and accuracy.

Efficiency and Speed

One of the most striking results is the massive speedup achieved by ETR. ETR-D is approximately 550x faster than Patch-NetVLAD-p and 271x faster than SP-SuperGlue. This efficiency allows ETR to be used in real-time scenarios where traditional re-ranking would be prohibitive.

Accuracy and Robustness

ETR consistently improves the performance of various global retrieval methods. On the MSLS challenge set, ETR-S achieved a Recall@1 of 53.9%, surpassing state-of-the-art methods like DELG local (52.2%) and Patch-NetVLAD (48.1%).

Performance evaluation of ETR-D on four major benchmarks, demonstrating consistent improvements over global retrieval baselines as the number of re-ranked candidates increases.

As shown in the evaluation across multiple datasets, ETR-D provides a substantial boost over baselines like NetVLAD, AP-GEM, and SFRS. The performance scales positively with the number of candidates re-ranked (top-30 to top-300), highlighting the model's ability to recover correct matches that were poorly ranked by global-only methods. Qualitative analysis further reveals that ETR is particularly robust to extreme environmental changes, such as day-to-night transitions.

Conclusion

ETR represents a significant step forward in making high-accuracy re-ranking practical for large-scale VPR. By leveraging the parallel processing power of Transformers, ETR achieves state-of-the-art accuracy while being orders of magnitude faster than traditional geometric verification. Future work will focus on further optimizing feature extraction and exploring end-to-end frameworks to enhance real-time performance even further.

Summary updated

Abstract

Visual place recognition is to estimate the geographical location of a given image, which is usually addressed by recognizing its similar reference images from a database. The reference images are usually retrieved via similarity search using global descriptor, and the local descriptors are used to re-rank the initial retrieved candidates. The local descriptors re-ranking can significantly improve the accuracy of global retrieval but comes at a high computational cost. To achieve a good trade-off between accuracy and efficiency, we propose an Efficient Transformer for Re-ranking (ETR), utilizing both global and local descriptors to re-rank the top candidates in a single shot. In contrast to traditional re-ranking methods, we leverage self-attention to capture relationships between local descriptors in a single image and cross-attention to explore the similarity of the image pairs. We show that the proposed model can be regarded as a general re-ranking algorithm for significantly boosting the performance of other global-only retrieval methods. Extensive experimental results show that our method outperforms state-of-the-arts and is orders of magnitude faster in terms of computational efficiency.
BibTeX
@inproceedings{zhang2023etr,
  author = {Hao Zhang and Xin Chen and Heming Jing and Yingbin Zheng and Yuan Wu and Cheng Jin},
  title = {{ETR: An Efficient Transformer for Re-ranking in Visual Place Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00562},
}
Figure