SuperGF: Unifying Local and Global Features for Visual Localization

Wenzheng Song, Ran Yan, Boshu Lei, Takayuki OkataniarXiv 2022 · DOI · arXiv

Abstract

Visual localization typically relies on a two-stage hierarchical pipeline: global image retrieval followed by local feature matching. However, maintaining separate models for these tasks is computationally expensive and redundant. We present SuperGF, a Transformer-based framework that generates global image descriptors by aggregating local features already extracted for matching. By bridging the feature-level gap between retrieval and matching, SuperGF achieves state-of-the-art localization accuracy while significantly reducing memory and computational overhead.

Introduction

Visual localization—the task of determining a camera's precise 6-Degree-of-Freedom (6-DoF) pose—is a cornerstone of robotics and autonomous systems. The standard approach involves a "hierarchical" process: first, image retrieval identifies candidate images from a database using global features; second, precise geometric correspondences are established using local features.

A significant bottleneck in this pipeline is the "feature-level gap." Retrieval requires high-level semantic understanding (e.g., identifying a street), whereas matching demands spatial invariance and pixel-level precision (e.g., identifying a specific corner). Consequently, most systems deploy two heavy, independent neural networks, such as NetVLAD for global and SuperPoint for local features. SuperGF addresses this redundancy by "upcycling" local features into a global descriptor using a Transformer architecture, treating global feature extraction as an aggregation problem rather than a separate vision task.

Methodology

SuperGF introduces a holistic model that leverages the global receptive field of Transformers to synthesize global context from local descriptors. The framework is proposed in two distinct architectural variants to suit different computational needs.

The SuperGF architecture features two versions: a Dense version operating on feature maps and a Sparse version utilizing Slot Attention Clustering on keypoints.

Dense and Sparse Architectures

  1. Dense Version: This variant operates on intermediate feature maps from a local feature encoder. A CNN-based tokenizer downsamples these maps before they are processed by Transformer encoder layers to capture long-range dependencies.
  2. Sparse Version: Designed for extreme efficiency, this version operates directly on sparse keypoints (descriptors, positions, and confidence scores). It employs Slot Attention Clustering to group a variable number of points into a fixed set of "cluster tokens," which are then refined by the Transformer.

Both versions conclude with Generalized Mean (GeM) pooling to produce the final global descriptor $\mathcal{D}$.

Training Strategy

SuperGF is trained using a Listwise Average Precision (AP) loss, which optimizes the ranking of an entire image batch directly. This approach offers more stability than traditional triplet losses by considering the global structure of the embedding space.

The training pipeline utilizes a Listwise AP loss to optimize the ranking of query images against positive and negative database samples.

For the sparse version, an additional attention decorrelation loss is implemented to ensure that different attention slots capture non-redundant information across the image, maximizing the information density of the global fingerprint.

Experiments and Results

The model was evaluated on standard benchmarks including MSLS, Pitts30k, and the RobotCar Seasons dataset, which includes challenging conditions like night, rain, and snow.

Performance and Efficiency

  • Localization Accuracy: On the challenging RobotCar Seasons "Night" sequence, SuperGF (paired with SuperGlue) achieved a pose recall of 34.2 / 70.3 / 93.0 at various distance/angle thresholds. This represents a massive improvement over the standard NetVLAD+SuperPoint baseline (6.6 / 17.1 / 32.2).
  • Computational Efficiency: The SuperGF-sparse model is remarkably lightweight, requiring only 2.2 MB of memory and 2.3 ms for feature extraction on an NVIDIA Titan RTX. This is nearly 70x smaller than competing state-of-the-art models like SFRS (149 MB).
  • Robustness: SuperGF showed particular strength in cross-condition matching, successfully localizing nighttime queries against daytime databases where traditional semantic-heavy models often fail.

Conclusion

SuperGF demonstrates that local features contain sufficient information for robust global retrieval when aggregated through a global receptive field. By eliminating the need for a separate global backbone, SuperGF provides a faster, more memory-efficient, and more accurate solution for real-time visual localization. Future work may explore end-to-end training of the local feature extractor specifically for this unified task to further optimize performance.

Summary updated

Abstract

Advanced visual localization techniques encompass image retrieval challenges and 6 Degree-of-Freedom (DoF) camera pose estimation, such as hierarchical localization. Thus, they must extract global and local features from input images. Previous methods have achieved this through resource-intensive or accuracy-reducing means, such as combinatorial pipelines or multi-task distillation. In this study, we present a novel method called SuperGF, which effectively unifies local and global features for visual localization, leading to a higher trade-off between localization accuracy and computational efficiency. Specifically, SuperGF is a transformer-based aggregation model that operates directly on image-matching-specific local features and generates global features for retrieval. We conduct experimental evaluations of our method in terms of both accuracy and efficiency, demonstrating its advantages over other methods. We also provide implementations of SuperGF using various types of local features, including dense and sparse learning-based or hand-crafted descriptors.
BibTeX
@misc{song2022supergf,
  author = {Wenzheng Song and Ran Yan and Boshu Lei and Takayuki Okatani},
  title = {{SuperGF: Unifying Local and Global Features for Visual Localization}},
  year = {2022},
  eprint = {2212.13105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.13105},
}
Figure