GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization

Vicente Vivanco Cepeda, Gaurav Kumar Nayak, Mubarak Shah

TL;DR

GeoCLIP is a novel framework that treats worldwide geo-localization as an image-to-GPS retrieval task rather than a standard classification problem. By aligning image features with a hierarchical, continuous coordinate encoder in a shared embedding space, it achieves state-of-the-art accuracy and remarkable data efficiency, maintaining high performance even when trained on just 5% of available data.

What Problem Does This Paper Address?

Determining exactly where a photo was taken—worldwide geo-localization—is a massive challenge due to the Earth's geographic diversity. Traditional approaches generally fall into two camps:

  1. Image-to-Image Retrieval: Matching a query photo against a massive gallery of geotagged images. This works for specific cities but is computationally impossible at a global scale.
  2. Classification: Dividing the Earth into discrete "cells" or bins. This is the current standard for global tasks, but it suffers from "boundary effects" (where an image near a cell edge is misclassified) and limited resolution dictated by the number of predefined classes.

The authors of GeoCLIP aim to bridge this gap by modeling the Earth as a continuous function, allowing for much finer localization without the constraints of discrete bins.

Methods

GeoCLIP uses a "two-tower" architecture inspired by CLIP, consisting of an image encoder and a novel location encoder.

  • Image Encoder: The model leverages a frozen, pre-trained CLIP ViT-L/14 backbone. Two additional trainable linear layers are added to adapt these general visual features to the specific task of geo-localization.
  • Location Encoder: This is the core innovation. To represent a 2D GPS coordinate $(lat, lon)$ as a high-dimensional feature, the authors use three key steps:
  • Equal Earth Projection (EEP): Converting coordinates to a projection that minimizes the distortion found in standard latitude/longitude systems (especially near the poles).
  • Random Fourier Features (RFF): Standard neural networks struggle to learn high-frequency spatial details (a phenomenon called "spectral bias"). RFF maps the 2D coordinates into a higher-dimensional space using sinusoidal functions, enabling the model to distinguish between nearby locations like different streets.
  • Hierarchical Representation: The encoder processes coordinates at multiple resolutions (coarse to fine) and aggregates them, allowing the model to understand both broad regions and specific landmarks simultaneously.
  • Contrastive Learning: The model is trained to maximize the similarity between an image's features and its corresponding GPS embedding while pushing away "negative" GPS samples. A dynamic queue of GPS coordinates is maintained during training to provide a rich set of negative examples without needing extra images.
Figure 1: The objective of image-based geo-localization is to find the GPS coordinate of a given image. (A) We formulate this problem as an image-to-GPS retrieval approach where the image is a query, and the gallery is a collection of GPS coordinates. (B) We propose a novel method, GeoCLIP, having two major components: a location encoder L (·) to obtain high-dimensional features from the 2D GPS coordinates and a CLIP-based image encoder V(·) to obtain the image features. We perform the matching of features of a query image against a gallery of GPS embeddings, and the most similar GPS embedding is the predicted GPS coordinates.

Datasets

  • Training: The MediaEval Placing Tasks 2016 (MP-16) dataset, containing 4.72 million geotagged images from Flickr.
  • Evaluation:
    • Im2GPS3k: A standard benchmark for geo-localization.
  • Google World Streets 15K (GWS15k): A newer, more challenging dataset with images uniformly sampled across the globe, reducing the "tourist hotspot" bias found in Flickr data.
  • YFCC26k: Used for additional validation.

Key Insights and Ideas

  • GPS as a Continuous Query: By encoding GPS coordinates directly, the model isn't limited to 20,000 fixed classes. It can theoretically match a query image to any point on the globe.
  • Overcoming Spectral Bias: The use of RFF is critical. The authors show that a simple MLP fails at fine-grained localization (1km scale), but adding RFF "nearly doubles" accuracy at that resolution.
  • Zero-Shot Text-to-Location: Because the image encoder is aligned with CLIP's text space, GeoCLIP can perform geo-localization using text queries (e.g., typing "Desert" generates a heatmap of desert regions worldwide) without any additional training.
  • Extreme Data Efficiency: The model is surprisingly robust. Training on only 20% of the data results in almost no performance loss, and even at 5%, it remains highly competitive.
Figure 3: We vary the amount of training data from 100% to as low as 5%. GeoCLIP achieves respectable accuracy even with only 5% of the training data, demonstrating its utility in limited data scenarios.

Key Results

GeoCLIP sets new state-of-the-art (SOTA) benchmarks across multiple datasets:

  • Im2GPS3k: Achieved 14.11% accuracy at the 1km (street) level, a significant improvement over the previous SOTA of 12.8%.
  • GWS15k: On this more difficult dataset, GeoCLIP nearly doubled the accuracy of previous models at the continent level (74.1% vs. 50.5%) and showed massive gains at the country level (45.7% vs. 26.9%).
  • Downstream Utility: The authors demonstrated that their pre-trained location encoder could be used to improve standard image classification tasks on the NUS-WIDE dataset, proving the features are semantically rich beyond just localization.

Limitations and Future Work

  • Computational Overhead: While training the GeoCLIP layers is fast, pre-computing CLIP features for millions of high-resolution images is time-consuming and requires significant storage.
  • Hyperparameter Sensitivity: The performance of the location encoder depends on the choice of frequency ranges ($\sigma$ values) in the RFF layers. While the hierarchical approach mitigates this, finding the optimal balance for all scales remains a challenge.
  • Future Directions: The authors suggest exploring the location encoder's utility in other location-aware tasks, such as autonomous driving or digital forensics.

Summary updated

Abstract

Worldwide Geo-localization aims to pinpoint the precise location of images taken anywhere on Earth. This task has considerable challenges due to immense variation in geographic landscapes. The image-to-image retrieval-based approaches fail to solve this problem on a global scale as it is not feasible to construct a large gallery of images covering the entire world. Instead, existing approaches divide the globe into discrete geographic cells, transforming the problem into a classification task. However, their performance is limited by the predefined classes and often results in inaccurate localizations when an image's location significantly deviates from its class center. To overcome these limitations, we propose GeoCLIP, a novel CLIP-inspired Image-to-GPS retrieval approach that enforces alignment between the image and its corresponding GPS locations. GeoCLIP's location encoder models the Earth as a continuous function by employing positional encoding through random Fourier features and constructing a hierarchical representation that captures information at varying resolutions to yield a semantically rich high-dimensional feature suitable to use even beyond geo-localization. To the best of our knowledge, this is the first work employing GPS encoding for geo-localization. We demonstrate the efficacy of our method via extensive experiments and ablations on benchmark datasets. We achieve competitive performance with just 20% of training data, highlighting its effectiveness even in limited-data settings. Furthermore, we qualitatively demonstrate geo-localization using a text query by leveraging CLIP backbone of our image encoder. The project webpage is available at: https://vicentevivan.github.io/GeoCLIP
BibTeX
@inproceedings{cepeda2023geoclip,
  author = {Vicente Vivanco Cepeda and Gaurav Kumar Nayak and Mubarak Shah},
  title = {{GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=I18BXotQ7j},
}
Figure