GEOCAPSNET: Ground to Aerial View Image Geo-Localization using Capsule Network

Bin Sun, Chen Chen, Yingying Zhu, Jianmin Jiang

GeoCapsNet: Aerial to Ground view Image Geo-localization using Capsule Network

TL;DR

GeoCapsNet introduces Capsule Networks to the challenge of cross-view geo-localization—matching ground-level photos to satellite imagery. By leveraging capsules to model spatial hierarchies and a novel "Soft-TriHard" loss function, the model achieves state-of-the-art accuracy while being significantly more compact and efficient than previous convolutional neural network (CNN) approaches.

What Problem Does This Paper Address?

Imagine you have a photo of a street corner but no GPS metadata. How do you find where it was taken? Traditional "ground-to-ground" matching works well in tourist-heavy cities but fails in rural or less-documented areas. Satellite imagery, however, covers nearly the entire planet.

Overview of the cross-view geo-localization task: matching a ground-level query to satellite imagery.

The catch? Matching a horizontal street-level view to a top-down aerial view is incredibly difficult. The viewpoint shift is drastic: a building looks like a rectangle from above but a facade from the ground. Standard CNNs often struggle with these geometric discrepancies because they tend to lose spatial relationship information during the feature extraction process, particularly during pooling layers.

Methods

The authors propose GeoCapsNet, a Siamese network architecture designed to bridge this domain gap.

The GeoCapsNet architecture, showing the Siamese structure with ResNetX feature extraction and Capsule layers.
  1. Feature Extraction (ResNetX): Both the ground and aerial branches start with a residual network (ResNetX) to extract basic visual features.
  2. Capsule Layers: This is the core innovation. Instead of standard neurons, the model uses "Capsules"—groups of neurons whose output is a vector. The length of the vector represents the probability of an entity existing, while its orientation encodes spatial properties like position and rotation. This allows the network to learn "parts-to-whole" relationships, which are more robust to viewpoint changes.
  3. Two Variants:
    • GeoCapsNet-I: Uses separate weights for the capsule layers in the ground and aerial branches.
  4. GeoCapsNet-II: Shares weights between the branches, forcing the model to learn a unified internal representation for both views.
  5. Soft-TriHard Loss: The authors introduced a weighted soft-margin triplet loss with online batch hard sample mining. During training, the model specifically hunts for the "hardest" negative samples—images that look similar but are at different locations—to sharpen its discriminative power.

Datasets

The model was evaluated on two major benchmarks:

  • CVUSA: A dataset featuring matching pairs of ground-level panoramas and satellite images. Panoramas provide a full 360-degree context, making this dataset relatively easier.
  • Vo and Hays: A more challenging dataset consisting of single-view street images and overhead photos from 11 U.S. cities, totaling over 1 million image pairs.

Key Insights and Results

  • Preserving Spatial Hierarchy: Geo-localization depends heavily on the relative positions of objects (trees, roads, buildings). Capsules preserve this geometry better than standard CNNs.
  • Weight Sharing Wins: Surprisingly, GeoCapsNet-II (shared weights) outperformed GeoCapsNet-I. This suggests a fundamental "semantic DNA" shared between aerial and ground views that a shared-weight capsule layer can capture.
  • Efficiency and Performance: GeoCapsNet set new performance benchmarks. On the CVUSA dataset, GeoCapsNet-II achieved a 98.07% Top-1% recall, significantly outperforming CVM-Net (91.40%). At the most difficult metric—Top-1 recall—it showed a 20.53% improvement.
  • Compactness: The model is much leaner, using ~65 million parameters compared to CVM-Net’s 160 million. The feature "code length" is also halved (2048 vs 4096), leading to faster retrieval and lower storage requirements.

Limitations and Future Work

Accuracy is lower on the Vo and Hays dataset compared to CVUSA because single-view images contain much less information than panoramas. The authors suggest that the geometric discrepancy between a single-view ground photo and a wide aerial shot remains a significant challenge. Future work will focus on better aligning the limited field of view of standard cameras with expansive satellite views.

Summary updated

Abstract

The task of cross-view image geo-localization aims to determine the geo-location (GPS coordinates) of a query ground-view image by matching it with the GPS-tagged aerial (satellite) images in a reference dataset. Due to the dramatic changes of viewpoint, matching the cross-view images is challenging. In this paper, we propose the GeoCapsNet based on the capsule network for ground-to-aerial image geo-localization. The network first extracts features from both ground and aerial images via standard convolution layers and the capsule layers further encode the features to model the spatial feature hierarchies and enhance the representation power. Moreover, we introduce a simple and effective weighted soft-margin triplet loss with online batch hard sample mining, which can greatly improve the image retrieval accuracy. Experimental results show that our GeoCapsNet significantly outperforms the state-of-the-art approaches on two benchmark datasets.
BibTeX
@inproceedings{sun2019geocapsnet,
  author = {Bin Sun and Chen Chen and Yingying Zhu and Jianmin Jiang},
  title = {{GEOCAPSNET: Ground to Aerial View Image Geo-Localization using Capsule Network}},
  booktitle = {ICME},
  year = {2019},
  url = {https://openreview.net/forum?id=dlIyYUVnii},
}
Figure