Interpretable Semantic Photo Geolocation

Jonas Theiner, Eric Müller-Budack, Ralph EwerthWACV 2022

TL;DR

This paper tackles the "black box" problem in planet-scale photo geolocation by introducing a novel semantic partitioning method that aligns geographical cells with real-world boundaries (cities, roads, etc.), making predictions more understandable. They also propose a "concept influence" metric to quantify how much specific visual concepts contribute to a model's geolocation guess, offering a new way to analyze and interpret existing models.

What Problem Does This Paper Address?

Estimating the precise geographic location of a photo solely from its visual content, known as planet-scale photo geolocalization, has seen impressive accuracy gains thanks to deep learning. However, these models often act as "black boxes," making it difficult for humans to understand why a particular prediction was made. This lack of interpretability hinders validation and trust. Current methods typically divide the Earth into arbitrary grid cells (partitions) for classification. The authors argue that these partitions are crucial for performance but also contribute to the opacity of the models. They aim to make these models more transparent by creating more meaningful partitions and by providing tools to analyze the influence of visual concepts on predictions.

Methods

The paper introduces two main contributions:

  • Semantic Partitioning (SemP): Instead of arbitrary grid cells, SemP creates partitions based on real-world geographic boundaries extracted from OpenStreetMap (OSM). These boundaries can be territorial (cities, countries), natural (rivers, mountains), or man-made (roads, railways). This approach aims to create cells that are more intuitively understandable and likely to share similar geographic attributes, leading to more interpretable predictions. The process involves reverse geocoding GPS coordinates to obtain address information, constructing a hierarchy of these locations, and then defining cells based on this semantic hierarchy.
  • Concept Influence Metric: This metric quantifies the importance of specific visual concepts (e.g., "building," "mountain," "person") for a given geolocation prediction. It works by combining an "explanation map" (which highlights important pixels for a prediction) with a "segmentation map" (which identifies regions of specific concepts in the image). By measuring the overlap between the top-k most important pixels and the pixels belonging to a concept, the metric provides a score indicating how influential that concept is for the prediction. This allows for post-hoc analysis of trained models.

Datasets

  • MediaEval Placing Task 2016 (MP-16): Used for partitioning construction and training. This dataset is a subset of YFCC100M and contains images with GPS coordinates, covering landmarks, landscapes, and images with limited geographical cues.
  • YFCC-Val26k: A randomly sampled subset of YFCC100M used for validation.
  • YFCC4K: A popular benchmark dataset for general computer vision tasks, used for testing.
  • Im2GPS and Im2GPS3k: Datasets containing landmarks and images recognizable in a generic sense, also used for testing.

Key Insights and Ideas

  • Real-world boundaries make better partitions: Aligning geolocalization cells with natural and man-made geographic features (like cities or roads) leads to more intuitive and understandable predictions compared to arbitrary grid cells.
  • Hierarchy matters, but not always as expected: While hierarchical partitioning has been shown to improve performance, the paper finds that for humans, a simpler, finest-level partitioning might be more interpretable than a complex, multi-level hierarchy.
  • Concept influence reveals what matters: The concept influence metric provides a quantitative way to understand which visual elements (e.g., "bridge," "building," "sky") are most important for a model's geolocation guess, offering valuable insights into the model's reasoning.
  • General concepts are less influential for precise localization: Concepts like "grass" or "water" have low influence scores at fine-grained distances (0-25 km), as they are visually similar globally. Conversely, concepts like "building" or "mountain" are more influential for precise localization.

Key Results

  • The proposed semantic partitioning method (SemP) achieves state-of-the-art geolocational accuracy on benchmark test sets, performing comparably to or better than existing methods, including those using the s2 partitioning.
  • When using the EfficientNet architecture, SemP demonstrates superior performance across various scales and test sets compared to state-of-the-art models.
  • The concept influence metric successfully identifies concepts that are more relevant for precise localization (e.g., "building," "mountain") versus those that are more relevant for broader estimations (e.g., "person," "tree").

Limitations and Future Work

  • Hierarchical interpretation: The paper notes that while hierarchical knowledge is generally beneficial, the specific hierarchical structure used in their SemP might not align with human perception of location hierarchy, leading to less interpretable results for complex hierarchies.
  • Future work: The authors plan to incorporate visual similarities between classes based on geographical features during optimization, potentially using knowledge bases. They also aim to explore how visual and spatial proximity of classes can be better leveraged.

Summary updated

BibTeX
@inproceedings{theiner2022interpretable,
  author = {Jonas Theiner and Eric Müller-Budack and Ralph Ewerth},
  title = {{Interpretable Semantic Photo Geolocation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure