Hierarchical Attention Fusion for Geo-Localization

Liqi Yan, Yiming Cui, Yingjie Victor Chen, Dongfang Liu

Abstract

Geo-localization is a critical task in computer vision. In this work, we cast the geo-localization as a 2D image retrieval task. Current state-of-the-art methods for 2D geo-localization are not robust to locate a scene with drastic scale variations because they only exploit features from one semantic level for image representations. To address this limitation, we introduce a hierarchical attention fusion network using multi-scale features for geo-localization. We extract the hierarchical feature maps from a convolutional neural network (CNN) and organically fuse the extracted features for image representations. Our training is self-supervised using adaptive weights to control the attention of feature emphasis from each hierarchical level. Evaluation results on the image retrieval and the large-scale geo-localization benchmarks indicate that our method outperforms the existing state-of-the-art methods.

Introduction: The Challenge of Scale

Image-based geo-localization aims to determine the geographic location of a query image by retrieving the most similar images from a large database of GPS-tagged photos. While treated as an image retrieval problem, the primary hurdle is scale variation. Landmarks can appear as dominant foreground structures or tiny, distant clusters of pixels. Traditional methods typically rely on a single semantic level—often the final layer of a CNN—which tends to lose fine-grained details during downsampling, leading to failures in complex urban environments where small details are often the only distinguishing features.

Methodology: Hierarchical Attention Fusion

To bridge this "semantic gap," the Hierarchical Attention Fusion (HAF) network utilizes a three-stage pipeline:

  1. Hierarchical Feature Extraction: Instead of relying on a single layer, the model extracts feature maps from three different stages of a VGG16 backbone (specifically layers Con3_2, Con4_3, and Con5_3). This captures a mix of low-level textures, mid-level shapes, and high-level semantic concepts.
  2. Attention Fusion Decoder: The model employs learnable attention masks that act like a spotlight, identifying which spatial regions at each hierarchical level are discriminative for location identification and which are merely noise (like cars or pedestrians).
  3. Weakly Supervised Training: The system is trained using a triplet ranking loss. It requires only GPS tags to identify "positive" (nearby) and "negative" (distant) image pairs, learning to minimize the distance between positives in the feature space.
The HAF architecture: Hierarchical features are extracted from a CNN backbone and processed through an Attention Fusion Decoder to create a robust image representation.

Key Insights and Results

Our evaluation across diverse benchmarks, including Pitts250k, Tokyo 24/7, and Oxford 5k, demonstrates that HAF consistently outperforms state-of-the-art methods like NetVLAD, CRN, and SuperPoint.

Filtering Environmental Noise

A significant advantage of the HAF method is its ability to ignore "distractor" objects. Qualitative heatmaps reveal that the attention mechanism effectively filters out transient elements such as cars, pedestrians, and vegetation, focusing instead on permanent structural landmarks that are useful for long-term localization.

Qualitative comparison of attention maps. Compared to NetVLAD, CRN, and SuperPoint, the HAF method focuses more accurately on stable, discriminative landmarks.

Performance Metrics

  • Image Retrieval: On the Oxford 5k benchmark, HAF achieved a 67.81 mAP, outperforming the next best method (CRN) by nearly 4%.
  • Geo-Localization: The model showed superior Recall@N across all large-scale datasets. In the Tokyo 24/7 dataset, the gap between HAF and previous methods was particularly pronounced, proving its robustness in dense urban settings.
  • Compactness: Despite using hierarchical features, the model produces a compact 256-dimensional representation, making it efficient for large-scale database searches.

Conclusion and Future Work

The HAF network demonstrates that multi-scale feature fusion is essential for robust geo-localization. By combining low-level spatial resolution with high-level semantic meaning, the model achieves state-of-the-art performance while remaining computationally efficient for retrieval. Future work will explore making hierarchical weights dynamically adaptive to individual input images and improving robustness against extreme seasonal changes.

Summary updated

Abstract

Geo-localization is a critical task in computer vision. In this work, we cast the geo-localization as a 2D image retrieval task. Current state-of-the-art methods for 2D geo-localization are not robust to locate a scene with drastic scale variations because they only exploit features from one semantic level for image representations. To address this limitation, we introduce a hierarchical attention fusion network using multi-scale features for geo-localization. We extract the hierarchical feature maps from a convolutional neural network (CNN) and organically fuse the extracted features for image representations. Our training is self-supervised using adaptive weights to control the attention of feature emphasis from each hierarchical level. Evaluation results on the image retrieval and the large-scale geo-localization benchmarks indicate that our method outperforms the existing state-of-the-art methods. Code is available here: \url{https://github.com/YanLiqi/HAF}.
BibTeX
@inproceedings{yan2021hierarchical,
  author = {Liqi Yan and Yiming Cui and Yingjie Victor Chen and Dongfang Liu},
  title = {{Hierarchical Attention Fusion for Geo-Localization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=wCtv3UJ58YJ},
}
Figure