Are These From the Same Place? Seeing the Unseen in Cross-View Image Geo-Localization

Royston Rodrigues, Masahiro TaniWACV 2021

TL;DR

This paper tackles the challenge of cross-view image geo-localization, specifically addressing how to match ground-level images to aerial imagery when scenes change over time (e.g., new buildings appear). They introduce a novel data augmentation technique that "hallucinates" unseen objects and a multi-scale attentive network to improve matching accuracy, achieving significant gains on standard benchmarks.

What Problem Does This Paper Address?

Digital maps and geo-tagged imagery are increasingly used for visual navigation. A key task is cross-view image geo-localization, where the goal is to determine the location of a ground-level image by matching it against a database of geo-tagged aerial imagery. While existing methods excel at matching images with similar appearances, they struggle when there are significant temporal differences between the ground and aerial views. Over time, buildings can be constructed, vegetation can change, or objects can disappear, creating a "temporal gap." Storing aerial imagery for every possible time instant is impractical, limiting the applicability of current geo-localization systems. This paper aims to bridge this temporal gap by enabling matching networks to handle unseen objects and scene variations.

Methods

The core of the proposed solution lies in a two-pronged approach:

  1. Semantically Driven Data Augmentation: To simulate the temporal gap, the authors introduce a novel data augmentation technique. They use a semantic segmentation module to identify objects in ground-view images and then create augmented samples by either "keeping" or "removing" specific object classes.
  2. "Keep" mode: Masks out regions of certain classes (like sky or car, which might not be consistently present in aerial views) to simulate their absence.
  3. "Remove" mode: Masks out regions of other classes to simulate their disappearance. This augmentation is applied only to the ground-view images, forcing the network to learn to match based on the remaining visible information and to "hallucinate" or infer the presence of missing objects in the aerial view.

  4. Multi-Scale Attentive Backbone Network: The augmented images are fed into a Siamese network architecture. The backbone of this network is a modified ResNet-18. Crucially, it extracts features from multiple intermediate layers of the ResNet, capturing information at different spatial scales. These multi-scale features are then processed by convolutional block attention modules (CBAMs) which apply attention along both spatial and channel dimensions. This allows the network to focus on salient features at each scale and maintain correspondences across views. The features from different scales are aggregated, and the entire network is trained using a contrastive loss function to bring matching pairs closer and push non-matching pairs apart.

Datasets

The proposed method is evaluated on two standard cross-view image geo-localization datasets:

  • CVACT [12]: Contains ground-view panoramas and corresponding aerial images.
  • CVUSA [26]: Similar to CVACT, with ground-view panoramas and aerial imagery.

The paper also mentions using a large-scale test split of CVACT for evaluation at scale.

Key Insights and Ideas

  • "Hallucinating" unseen objects is key: The most significant insight is that explicitly training the network to handle missing or newly appeared objects through semantic augmentation is crucial for robust geo-localization in temporally varying scenes.
  • Multi-scale attention is beneficial: Capturing features at different spatial resolutions and applying attention mechanisms helps in robustly matching corresponding regions across vastly different viewpoints.
  • Complementarity with existing methods: The proposed method is shown to be complementary to existing techniques like SAFA [18]. By fusing their retrieval scores, significant performance boosts are achieved, suggesting that the proposed method addresses a different aspect of the problem (temporal variation) than SAFA (matching visible objects).

Key Results

The paper reports substantial improvements in top-1 recall rates:

  • CVUSA: The proposed method achieves a top-1 recall of 75.95%, an increase from 61.24% without augmentation. When fused with SAFA [18], the top-1 recall reaches 93.09%, up from 89.84%.
  • CVACT-val: The method achieves a top-1 recall of 73.19%. When fused with SAFA [18], the top-1 recall improves to 87.21%, up from 81.03%.
  • Qualitative results (Figure 8) demonstrate the method's ability to match unseen image regions (e.g., newly constructed buildings) and handle shadows, where SAFA [18] sometimes fails.

Limitations and Future Work

  • Data Augmentation Asymmetry: The semantic augmentation is applied only to ground-view images. The authors note that a similar approach for aerial images was not pursued due to the lack of suitable off-the-shelf segmentation modules for those datasets.
  • Failure Cases: The paper acknowledges failure cases, particularly "excess hallucination" where the method might over-augment when objects are clearly visible in both views. In such scenarios, methods like SAFA [18] that focus on visible objects perform better.
  • Future Work: While not explicitly detailed, the success of the complementary approach suggests future work could explore more sophisticated fusion strategies or methods that can dynamically adapt their augmentation based on scene visibility. The authors also highlight the importance of multi-scale attention and suggest further exploration of its benefits.

Summary updated

BibTeX
@inproceedings{rodrigues2021are,
  author = {Royston Rodrigues and Masahiro Tani},
  title = {{Are These From the Same Place? Seeing the Unseen in Cross-View Image Geo-Localization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure