AcrossPlace: Do Vision--Language Models Recognize Places Beyond Their Source Views? A Comprehensive Benchmark

Zhenyu Li, Tianyi Shang, Shuhan Dong, Ruotong Xu, Qingyu ZhuarXiv 2026 · arXiv

TL;DR

AcrossPlace is a new, provenance-aware benchmark designed to reveal whether vision-language models (VLMs) can recognize geographic locations across different viewpoints, or if they are simply matching text to specific image features. The researchers find a significant "observation-transfer gap," where models that successfully match a description to its source image often fail when asked to find the same place in a completely different photograph.

What Problem Does This Paper Address?

Imagine you describe a specific house: "Wooden siding with a gabled roof and an attached garage." A VLM might easily find the exact photo used to write that description. But can it find another photo of the same house taken from the opposite side of the street three years later?

Current benchmarks for language-guided place recognition often entangle three distinct challenges:

  1. Observation Transfer: Recognizing a place across different views/times.
  2. Gallery Competition: Distinguishing a target from a sea of similar-looking distractors.
  3. Linguistic Reliability: Ensuring the text description actually contains useful, distinctive information.

The authors argue that existing evaluations often hide a model's inability to generalize by allowing it to "cheat" by matching descriptions back to their original source images.

Figure 1 illustrates the difference between successful and failed place retrieval when moving beyond the source image.

Methods

The core innovation of AcrossPlace is its "source-target independence." Every description in the dataset is linked to a specific "source" image, but the retrieval task is strictly performed on "target" images—disjoint observations of the same place.

The authors introduced three controlled diagnostic evaluations:

  • D1 (Observation Transfer): Compares retrieval performance on the original source images versus independent target views.
  • D2 (Gallery Competition): Measures how performance scales as the number of candidate places in the "gallery" increases.
  • D3 (Evidence Reliability): A human audit of 240 descriptions to check if the AI-generated text is actually supported by the visual evidence.

They evaluated several state-of-the-art models, including CLIP, SigLIP2, and BLIP, using both zero-shot (pretrained) and place-supervised (fine-tuned) configurations.

Datasets

AcrossPlace is built upon the GSV-Cities dataset. Its vital statistics include:

  • 56,726 unique places across 23 cities.
  • 156,800 descriptions automatically generated by Qwen3-VL and filtered by rules.
  • 113,404 target images for retrieval.

The benchmark uses two primary protocols:

  • P0 (Within-city): Tests on unseen places within cities the model saw during training.
  • P1 (City-holdout): Tests generalization to entirely new cities.
Figure 2 provides an overview of the AcrossPlace construction and the bidirectional retrieval task.

Key Insights and Ideas

  • The Observation-Transfer Gap: There is a massive drop in performance when a model moves from "image matching" (source) to "place recognition" (target). This suggests models are often overfitting to transient image features (like a specific car parked out front) rather than learning the enduring identity of a location.
  • Competition is the Killer: Place recognition isn't just about finding a match; it's about not picking the wrong one. As the gallery size grows, performance plummets, highlighting that "perceptual aliasing" (different places looking similar) is a major hurdle.
  • LLM Hallucinations in Captions: The human audit revealed that automated screening doesn't always catch factual errors. Some descriptions were "rule-valid" but contained details not visible in the images, while others were "rule-uncertain" but perfectly accurate.

Key Results

  • Performance Drop: For a place-supervised SigLIP2 model on the P0 protocol, Text-to-Image (T2I) Recall@1 dropped from 19.69% on source images to just 12.31% on independent target views.
  • Scaling Effects: On the P1 protocol, expanding the gallery from 500 places to 8,425 places caused Recall@1 to crash from 14.67% to a meager 2.87%.
  • Supervision Gains: While fine-tuning on place-specific data significantly improves performance over zero-shot models, it does not close the observation-transfer gap.
Figure 4 quantifies the performance degradation when switching from source to target views.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Geographic Scope: While 23 cities is a start, it doesn't cover the full diversity of global urban and rural environments.
  • Annotation Depth: The human audit was conducted by a single reviewer on a subset of the data; larger-scale, multi-reviewer validation would strengthen the findings.
  • Observation Conditions: Future versions could explicitly test more extreme changes in weather, season, or lighting.

Ultimately, AcrossPlace sets a new, more rigorous bar for the field, moving us closer to AI that can navigate and recognize the world as we do—not just by matching pixels, but by understanding identity.

Summary updated

Abstract

Language-guided place recognition requires recovering geographic identity across visual observations beyond the image from which a description originates. However, aggregate retrieval scores often entangle observation transfer, gallery competition, and linguistic evidence, obscuring the underlying recognition capability. We introduce AcrossPlace, a provenance-aware benchmark establishing a systematic evaluation standard for bidirectional language--vision place recognition. Built upon GSV-Cities, AcrossPlace comprises 56,726 places across 23 cities, 156,800 automatically generated, rule-filtered descriptions, and 113,404 target images. Each description retains its source-image provenance, while retrieval operates on disjoint target observations. Text-to-image (T2I) and image-to-text (I2T) retrieval share a unified place-identity relevance criterion supporting multiple valid matches. Two geographic protocols evaluate unseen places within represented cities (P0) and generalization to held-out cities (P1). Paired observation comparisons, gallery-controlled evaluations, fixed-query scaling, and blinded annotation auditing systematically characterize retrieval performance. Across three training seeds, place-supervised SigLIP2 T2I Recall@1 decreases from 19.69% on source images to 12.31% on independent targets under P0. For fixed P1 queries, expanding the gallery from 500 to 8,425 places reduces Recall@1 from 14.67% to 2.87%. A 240-description audit further reveals discrepancies between automated acceptance and factual evidence. These findings establish AcrossPlace as a reproducible evaluation standard for assessing transferable place identity across visual observations and geographic environments.
BibTeX
@misc{li2026acrossplace,
  author = {Zhenyu Li and Tianyi Shang and Shuhan Dong and Ruotong Xu and Qingyu Zhu},
  title = {{AcrossPlace: Do Vision--Language Models Recognize Places Beyond Their Source Views? A Comprehensive Benchmark}},
  year = {2026},
  eprint = {2611.15314},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15314},
}
Figure