Cross-Attention Between Satellite and Ground Views for Enhanced Fine-Grained Robot Geo-Localization

Dong Yuan, Frédéric Maire, Feras DayoubWACV 2024 · OpenReview

TL;DR

This paper introduces a novel Transformer-based architecture that uses cross-attention to fuse satellite and street-view images for more accurate robot localization. By replacing traditional correlation methods with attention mechanisms and combining classification with regression, the model significantly reduces localization errors, especially in challenging "unseen" areas, demonstrating improved generalization.

What Problem Does This Paper Address?

Robots navigating outdoors often rely on GPS, but GPS signals can be noisy or unavailable. A common approach to improve localization is "cross-view image geo-localization," where a robot's street-view image is matched with a GPS-tagged satellite image. Existing methods typically find the correct satellite image patch and use its center as a coarse location estimate. However, this "coarse location estimation" can still be off by tens of meters because street-view images aren't always captured precisely at the center of the satellite patch. This paper tackles the problem of "fine-grained localization," which aims to pinpoint the robot's exact pixel location within a known satellite image patch, going beyond just identifying the correct patch. Previous fine-grained methods often use simple correlation operations between image features, which can lose important details and limit the model's ability to generalize to new environments.

Methods

The core of this work is a novel cross-view feature fusion network built upon the Transformer architecture. Instead of relying on traditional correlation operations to find correspondences between street-view and satellite image features, the authors propose using self-attention and cross-attention layers.

  • Local Feature Extraction: Two modified ResNet50 networks are used to extract features from ground-view and satellite images independently.
  • Cross-view Feature Fusion Network: This is the heart of the method.
  • Self-Attention Blocks (SABs): These blocks enhance the features within each view (ground or satellite) by allowing them to attend to themselves, capturing richer internal representations.
  • Cross-Attention Blocks (CABs): These are the key innovation. They allow features from one view (e.g., street-view) to query features from the other view (e.g., satellite), and vice-versa. This mechanism enables the model to learn complex relationships and dependencies between local features across the two different perspectives, effectively fusing information and establishing correspondences. This is a significant departure from simpler correlation methods that treat features more independently.
  • Prediction Headers: To achieve fine-grained localization, the fused features are fed into two branches:
  • Multi-class Classification Header: This predicts which grid cell within the satellite image the street-view image likely belongs to.
  • Coordinate Offset Regression Header: This predicts a precise offset within the selected grid cell, refining the location estimate. This "coarse-to-fine" approach is designed to improve accuracy.

The authors also mention using Adaptive Sharpness-Aware Minimization (ASAM) for optimization, which helps prevent overfitting, a common issue with Transformer models.

Datasets

The proposed method is evaluated on two established benchmarks:

  • VIGOR: This dataset is specifically designed for cross-view fine-grained localization and features ground-view panoramas that can be captured anywhere within a satellite image patch, making it more realistic than previous benchmarks. It includes data from four cities (New York City, San Francisco, Chicago, Seattle). The paper uses both "same-area" (training and testing on data from the same cities) and "cross-area" (training on some cities, testing on others) protocols to evaluate generalization.
  • Oxford RobotCar: This dataset provides extensive sensor data from traversals of a route in Oxford, offering varied conditions.

Key Insights and Ideas

  • Cross-Attention for Richer Correspondences: The core insight is that cross-attention, by allowing features from one view to query and weigh features from another, can capture much more complex and nuanced relationships than simple correlation operations. This is crucial for accurately aligning street-view and satellite imagery, especially when visual cues might be subtle or partially obscured.
  • Coarse-to-Fine Localization is Effective: Combining a classification step (predicting the grid) with a regression step (predicting the offset within the grid) provides a more robust and accurate localization mechanism. This hierarchical approach helps to break down the complex localization problem into more manageable sub-problems.
  • Generalization to Unseen Areas: The Transformer architecture, with its attention mechanisms, appears to be inherently better at generalizing to new environments (cross-area testing) compared to previous methods. This is a critical finding for real-world robot deployment.

Key Results

  • VIGOR Benchmark: The proposed method significantly outperforms the state-of-the-art baseline (MCC [28]) on the VIGOR dataset.
  • In the "same-area" setting, it achieved a 43% reduction in median localization distance error for Pos+Semi-Pos testing.
  • In the more challenging "cross-area" setting, it achieved a 50% reduction in median localization distance error for Pos+Semi-Pos testing.
  • Oxford RobotCar Dataset: The model achieved competitive results, demonstrating its effectiveness even with less data and front-view images (which contain less information than panoramas).
  • Ablation Study: Removing the ASAM optimization slightly increased errors, but the core cross-attention method still significantly outperformed the baseline, highlighting the strength of the proposed fusion approach.

Limitations and Future Work

  • Front-view Image Information: The authors note that front-view images in the Oxford RobotCar dataset contain less information than panoramas, which can limit the effectiveness of cross-attention.
  • Dataset Size: The Oxford RobotCar dataset is smaller than VIGOR, which may hinder the training of Transformer models.
  • Future Work: The authors suggest exploring:
  • Utilizing multi-view street images to enhance performance.
  • Handling sequences of ground-view images as input.
  • Fine-grained orientation estimation in addition to location.

Summary updated

Abstract

Cross-view image geo-localization aims to determine the locations of outdoor robots by mapping current street-view images with GPS-tagged satellite image patches. Recent works have attained a remarkable level of accuracy in identifying which satellite patches the robot is in, where the location of the central pixel within the matched satellite patch is used as the robot coarse location estimation. This work focuses on robot fine-grained localization within a known satellite patch. Existing fine-grain localization work utilizes correlation operation to obtain similarity between satellite image local descriptors and street-view global descriptors. The correlation operation based on liner matching simplifies the interaction process between two views, leading to a large distance error and affecting model generalization. To address this issue, we devise a cross-view feature fusion network with self-attention and cross-attention layers to replace correlation operation. Additionally, we combine classification and regression prediction to further decrease location distance error. Experiments show that our novel network architecture outperforms the state-of-the-art, exhibiting better generalization capabilities in unseen areas. Specifically, our method reduces the median localization distance error by 43% and 50% respectively in the same area and unseen areas on the VIGOR benchmark.
BibTeX
@inproceedings{yuan2024cross,
  author = {Dong Yuan and Frédéric Maire and Feras Dayoub},
  title = {{Cross-Attention Between Satellite and Ground Views for Enhanced Fine-Grained Robot Geo-Localization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=faOUmppk9N},
}
Figure