Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales

Abstract

Vision-language models (VLMs) have advanced rapidly, yet their capacity for image-grounded geolocation in open-world conditions has not been comprehensively evaluated. We present EarthWhere, a comprehensive benchmark for VLM image geolocation that evaluates visual recognition, step-by-step reasoning, and evidence use. EarthWhere comprises 810 globally distributed images across two complementary scales: WhereCountry (broad recognition) and WhereStreet (fine-grained reasoning). We propose novel metrics, including a Shapley-reweighted thinking score, to attribute credit to intermediate reasoning steps. Benchmarking 13 state-of-the-art VLMs reveals that while proprietary models like Gemini-2.5-Pro lead, significant challenges remain in tool-use efficiency and mitigating regional biases.

Introduction: The Geolocation Challenge

Image geolocation—identifying the precise location of a photograph based on visual cues like architecture, vegetation, or signage—is a high-stakes task with applications in search-and-rescue and environmental monitoring. However, existing benchmarks often suffer from two major flaws: they prioritize the final coordinate over the reasoning process, and they evaluate models in isolation without access to external tools like web search.

EarthWhere addresses these gaps by shifting the focus toward "faithful" reasoning. It challenges models to not only find the right spot but to do so using the correct evidence, mirroring the multi-step process used by human experts.

The EarthWhere Benchmark

The EarthWhere dataset consists of 810 globally distributed images categorized into two tasks:

• WhereCountry: 500 multiple-choice questions using 360° panoramas to test broad, recognition-driven country identification.
• WhereStreet: 310 reasoning-heavy tasks sourced from Bilibili and YouTube, requiring fine-grained street-level identification. These tasks include human-verified reasoning chains and optional hints to simulate real-world challenges.

Expand

Source, p. 4

Methodology: Evaluating the Reasoning Chain

To move beyond simple distance-based accuracy, the researchers introduced several novel evaluation techniques:

1. Hierarchical Path Score (HPS): Models receive partial credit for identifying correct higher-level location prefixes (e.g., Country and Province) even if the final street address is incorrect.
2. Shapley-reweighted Thinking Score: This metric identifies 861 "key visual clues" and uses Shapley values to weigh each clue based on its marginal contribution to the correct answer. This allows for scoring the model's internal "thinking" trace.
3. Tool-Augmented Evaluation: Models are benchmarked both with and without native web search capabilities to assess how they utilize external information.

Expand

Source, p. 1

Key Results and Insights

Benchmarking 13 state-of-the-art VLMs yielded several critical insights:

• The "Overthinking" Trap: In broad recognition tasks (WhereCountry), providing models with more thinking time or web search access often decreased performance. For instance, GPT-4o's accuracy dropped by 13.2% with search, suggesting that internal weights are often more reliable for general recognition than noisy external data.
• Web Search as a Double-Edged Sword: While search hindered broad recognition, it provided a 6.5% relative boost in fine-grained street-level tasks where specific clues like signs are present. However, vague clues often led to "hallucinated" reasoning.
• Performance Gap and Regional Bias: Gemini-2.5-Pro emerged as the top performer with 56.32% accuracy. The best open-weight model, GLM-4.5V, reached 34.71%, highlighting a significant intelligence gap. Furthermore, models performed 42.7% better on Western-centric (YouTube) samples than on Chinese-centric (Bilibili) samples, revealing a deep-seated regional bias in training data.

Conclusion and Future Work

While current VLMs show promise, they remain far from human-level precision in fine-grained localization. Bottlenecks in tool-use—such as early termination of searches or failure to iterate—persist. Future research must focus on developing specialized geolocation tools and mitigating regional biases to ensure robust, global performance. The EarthWhere benchmark is open-sourced to facilitate these advancements.

@misc{qian2025where,
  author = {Zhaofang Qian and Hardy Chen and Zeyu Wang and Li Zhang and Zijun Wang and Xiaoke Huang and Hui Liu and Xianfeng Tang and Zeyu Zheng and Haoqin Tu and Cihang Xie and Yuyin Zhou},
  title = {{Where on Earth? A Vision-Language Benchmark for Probing Model Geolocation Skills Across Scales}},
  year = {2025},
  eprint = {2510.10880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10880},
}