AstroLoc: Robust Space to Ground Image Localizer

Gabriele Moreno Berton, Alex Stoken, Carlo Masone

TL;DR

This paper introduces AstroLoc, a novel system for accurately geolocating astronaut photos by leveraging millions of existing astronaut images for training, a significant departure from prior methods that only used satellite imagery. AstroLoc achieves state-of-the-art performance in Astronaut Photography Localization (APL) and demonstrates impressive robustness on related tasks like "lost-in-space" satellite localization.

What Problem Does This Paper Address?

Astronauts aboard the International Space Station (ISS) capture a vast number of Earth images daily, offering unique perspectives and high resolution. These images are invaluable for applications like climate science and disaster management. However, manually geolocating these photos is an extremely time-consuming and labor-intensive process. While automated methods using image retrieval have emerged, they typically rely solely on satellite imagery for training. This is a critical limitation because astronaut photos have distinct characteristics (oblique views, varied illumination, higher resolution) that differ from satellite imagery. The paper addresses this gap by proposing a method that effectively utilizes the abundant, yet weakly labeled, astronaut photo dataset to train a more robust and accurate localization model.

Methods

The core of AstroLoc's innovation lies in its training methodology, which leverages both astronaut and satellite imagery.

  1. Automated Weak Labeling and Footprint Estimation: The authors first process 300,000 manually "weakly labeled" astronaut photos (containing only a single geographic point) to automatically derive precise "full footprint" annotations (the coordinates of the image's four corners). This is achieved by using image matching techniques to find the extent of the astronaut photo within the satellite imagery database.
  2. Query-Satellite Pairwise Loss: A contrastive loss function is employed, pairing astronaut photos with their corresponding satellite counterparts. This loss encourages the model to learn similar representations for images depicting the same location, despite originating from different domains (astronaut vs. satellite).
  3. Unsupervised Mining (MUM Loss): To fully exploit the massive satellite image database, the authors introduce a novel "Unsupervised Mining" technique. This involves clustering satellite images and then sampling these clusters based on the distribution of astronaut photo queries. This ensures that the training data (drawn from satellite images) is more representative of the types of scenes astronauts photograph, leading to more effective learning. This mined data is then used with a Multi-Similarity loss.
  4. Combined Loss: The final AstroLoc model is trained using a combination of the pairwise loss and the unsupervised mining loss.

Datasets

  • Astronaut Photos: 300,000 manually weakly-labeled astronaut photos from the ISS were used for training. These were automatically processed to obtain full footprint annotations.
  • Satellite Imagery: A large database of geo-referenced satellite images (Sentinel-2 mosaics) was used for both training and as the search database. The paper mentions using zoom levels 8-12 for training and a worldwide database for testing.
  • Existing APL Datasets: The paper re-evaluates existing test sets from previous work (e.g., Texas, Gobi, Amazon) and introduces new, extended versions ("-L" versions) that include a wider range of query areas to better reflect real-world scenarios.
  • Lost-in-Space Dataset: Images from McCleary et al. [23] were used to test AstroLoc's performance on this related task.
  • Historical Space Shuttle Imagery: A dataset of 704 digitized film photographs from the early Space Shuttle era was used to evaluate robustness to domain changes.

Key Insights and Ideas

  • Leveraging Astronaut Data is Crucial: The paper strongly argues that training APL models solely on satellite imagery is a significant limitation. Incorporating astronaut photos, even with their inherent challenges (weak labels, domain differences), leads to substantial performance gains.
  • Automated Footprint Estimation is Key: The ability to automatically derive precise geographic footprints from weakly labeled astronaut photos is a critical enabler for using this data in metric learning frameworks.
  • Unsupervised Mining for Data Distribution Alignment: The novel unsupervised mining technique effectively bridges the gap between the distribution of available satellite imagery and the specific types of scenes captured by astronauts, leading to more relevant training data.
  • Robustness Across Domains: AstroLoc demonstrates remarkable adaptability, performing well not only on its primary task but also on significantly different tasks like historical imagery localization and "lost-in-space" scenarios without fine-tuning.

Key Results

  • State-of-the-Art APL Performance: AstroLoc achieves a staggering 35% average improvement in recall@1 over previous state-of-the-art methods.
  • Near-Perfect Recall: On existing APL datasets, AstroLoc consistently achieves recall@100 above 99%.
  • Improved Performance on Extended Datasets: On the newly introduced, more comprehensive test sets, AstroLoc maintains recall@100 above 96%.
  • Strong Performance on Related Tasks:
    • Lost-in-Space: Achieves over 50% recall@1, outperforming other methods by at least 45%.
  • Historical Imagery: Shows robustness to domain changes, with minimal performance drop compared to APL tasks.
  • Efficiency: AstroLoc uses a DINO-v2-base backbone with a linear layer, resulting in a model that is over 10 times lighter than previous methods like AnyLoc while achieving superior performance.

Limitations and Future Work

  • Failure Modes: The paper identifies several failure modes:
  • Forested Areas: Similar visual characteristics can lead to confusion with other forested regions.
  • Coastal Regions: Images with minimal landmass can result in retrieval of predominantly water-based satellite images.
  • Occlusions: Clouds can obscure significant portions of the image, hindering accurate localization.
  • Future Work: The authors suggest that training specifically to ignore cloud regions could improve performance in cases of occlusion. They also propose that hard negative mining might help disambiguate similar forested regions. Further exploration of these areas is planned.

Summary updated

Abstract

Astronauts take thousands of photos of Earth per day from the International Space Station, which, once localized on Earth's surface, are used for a multitude of tasks, ranging from climate change research to disaster management. The localization process, which has been performed manually for decades, has recently been approached through image retrieval solutions: given an astronaut photo, find its most similar match among a large database of geo-tagged satellite images, in a task called Astronaut Photography Localization (APL). Yet, existing APL approaches are trained only using satellite images, without taking advantage of the millions open-source astronaut photos. In this work we present the first APL pipeline capable of leveraging astronaut photos for training. We first produce full localization information for 300,000 manually weakly labeled astronaut photos through an automated pipeline, and then use these images to train a model, called AstroLoc. AstroLoc learns a robust representation of Earth's surface features through two losses: astronaut photos paired with their matching satellite counterparts in a pairwise loss, and a second loss on clusters of satellite imagery weighted by their relevance to astronaut photography via unsupervised mining. We find that AstroLoc achieves a staggering 35% average improvement in recall@1 over previous SOTA, pushing the limits of existing datasets with a recall@100 consistently over 99%. Finally, we note that AstroLoc, without any fine-tuning, provides excellent results for related tasks like the lost-in-space satellite problem and historical space imagery localization.
BibTeX
@inproceedings{berton2025astroloc,
  author = {Gabriele Moreno Berton and Alex Stoken and Carlo Masone},
  title = {{AstroLoc: Robust Space to Ground Image Localizer}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure