Vision-language models, which answer questions about images, do noticeably worse on satellite and street-level imagery from Nairobi or Mumbai than from New York or London. The usual reading is that they see and reason less well in places that are thin on training data. This proposal tests a less comfortable alternative. The places where models do well are also the places most thoroughly documented online, through census tables, Wikipedia pages, real-estate listings and countless geotagged photos. Some of the "reasoning" measured there may simply be recall of facts about a named or recognizable place. On top of that, the automatic labels used to score models in sparse regions come from crowdsourced maps that are themselves incomplete, so part of the measured Global South deficit may be wrong answer keys rather than wrong models.
We propose to build a geospatial reasoning benchmark that splits the North–South gap into three parts: a recall component, a perception component and a reasoning component. Questions are generated by programs from a structured description of each map tile, called a scene graph, which records roads, buildings, land use, elevation and spatial relations. Humans do not write questions. Annotators from each region instead correct that scene graph before any answer is computed. Each question is then posed three ways: as an image with the location revealed, as an image with location cues stripped, and as text with the verified scene graph and no image. The differences between these conditions, compared across matched tiles in dense and sparse regions, show how much of the gap each source explains. They also show how much a fully automated pipeline would have misstated it.
Core Claim
Our hypothesis has two parts, and the study can refute either. The first is that a measurable share of the Global North advantage reported by CityLens comes from place-specific recall. The CityLens benchmark evaluates 17 vision-language models on socioeconomic indicators across 17 cities. Its indicators, such as GDP, house price and building height, are exactly the kind of facts a model could have read about a well-known city. The second part is that automatically generated geospatial labels have higher error rates in sparse regions. If so, fully automated benchmarks overstate how badly models do there. The new element is a benchmark whose design controls the recall channel and audits label quality by region, so that a regional gap can be attributed to a cause rather than only observed.
How This Relates to Prior Work
Several papers motivate the recall worry. GeoChain builds a 21-step geolocation chain of thought on Mapillary street images, and its authors acknowledge that models may have seen these scenes during pre-training. More broadly, Back to Square One shows that web exposure to the Winograd Schema lets models bypass commonsense reasoning. Snoopy shows that few-shot accuracy tracks how often terms appear in pre-training data. No one has asked whether this frequency effect is geographically skewed. That is our question, because online documentation is strongly skewed toward cities and wealthy regions, as Not at Home on the Range and Fine-Grained Land Use Classification at the City Scale document for peer-produced maps and photo collections.
Several papers supply the method. DyVal generates fresh reasoning problems from directed acyclic graphs at controlled depth. Models that scored well on static benchmarks scored near zero on its problems. Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution, known as KBE-DME, extends this idea to visual question answering by re-sampling knowledge-graph triplets. We take the idea of generating questions from a graph at controllable depth. We differ in two ways. Our graph describes the physical content of a tile rather than open-domain knowledge, and every answer is computed by a program from that graph rather than written by a model. DyVal is text-only, while KBE-DME uses generic photos and lets a model produce the evaluation items. ToolQA supplies the pattern of human-designed templates filled automatically from a database. Synthesize Step-by-Step shows that turning a chart into executable code lets answers be computed exactly. We apply the same idea to map tiles.
The perception-versus-reasoning split comes from GeoSQA. Thirty geography students transcribed exam diagrams into text, and text models still scored near chance, with the best at 26.22% on four-way multiple choice. This showed that removing the vision step does not fix reasoning. CityLens finds a similar pattern from the other side: models work better as "eyes" that score visual attributes, which a separate regression then turns into predictions, than as "calculators" that output the number directly. Our text-only scene-graph condition carries the GeoSQA idea over to modern vision-language models and to real geography, and lets us measure it separately by region.
Two papers constrain the design. OSMDA synthesizes more than 200,000 satellite captions from OpenStreetMap without human annotation. Its authors concede that sparse map coverage degrades caption quality in exactly those regions. That is the label-error asymmetry we propose to measure rather than assume. Beyond Memorization found that cloze questions show performance decay after the model's training cutoff, while synthesized multi-step questions do not. In its influence analysis, the source paper was the most influential training document 77.5% of the time for cloze questions and 17.5% for synthesized ones. One implication is that "use only post-cutoff imagery" is not a clean contamination test. Another is that composed multi-step questions already resist recall, so our recall-blocking manipulation must be shown to add something beyond composition alone.
The human role follows the author–reviewer pattern. In Building a Role Specified Open-Domain Dialogue System, human review of model drafts was about 13 times more efficient than writing from scratch. Here, however, humans review the perception layer, the scene graph, rather than finished questions. Reviewing a graph needs local knowledge but no expertise in spatial reasoning. That sidesteps the problem of needing expensive experts to check complex reasoning.
Research Program
The first component is the scene-graph builder. For each tile, we merge OpenStreetMap features, a public elevation model and detections from an off-the-shelf remote-sensing detector. Mapillary street views are attached where they exist. The result is a typed graph with nodes such as buildings, road segments, water bodies and land-use parcels, and relations such as adjacency, connectivity, containment, distance and relative elevation. Annotators recruited from each sampled region correct this graph in a lightweight editor by adding missing buildings, deleting phantom roads and fixing land-use types. Every edit is logged, which gives a per-region measure of label error before verification.
The second component is the question generator. Human-written templates define composable operations: count, filter by type, shortest path on the road graph, compare elevations, compute area ratios and find the nearest feature. Questions are produced by chaining these operations to a chosen depth, as in DyVal, so difficulty can be controlled. Every answer is computed by the program from the verified graph. To counter template rigidity, a language model paraphrases question wording, and a human checks a sample of paraphrases for meaning drift.
The third component is the set of presentation conditions. In the location-revealed condition, the model sees the tile together with its place name and neighborhood. In the location-blocked condition, it sees the same tile rotated, with burned-in text and watermarks removed, and with no name. In the graph-only condition, it receives the verified scene graph as text and no image. We also split question types into two families. Recall-blocked questions have answers fixed entirely by the tile's local configuration, such as "how many buildings lie within 50 m of the river and uphill of the main road". Facts like this are essentially never written down online. Recall-open questions ask for documented place attributes of the CityLens kind.
Accuracies are then compared on tiles from Global North and Global South regions, matched by land-use type. The recall share of the gap is how much more the North gains than the South when the location is revealed. The perception share is how much the gap shrinks when the image is replaced by the verified graph. The residual gap in the graph-only condition is the reasoning share. Each share is reported with bootstrap confidence intervals computed over tiles.
Sampling covers 12 of the CityLens cities, split evenly between North and South, plus rural tiles in the same countries. Tiles are matched on land-use mix and building density, so that differences in what the scenes contain do not masquerade as regional bias. The planned scale is 1,500 tiles and roughly 15,000 questions, at depths from one to five operations.
Baselines
The naive reference point is the score of predicting the majority class or the training-set mean, together with the chance rate for multiple-choice items. This matters because CityLens reports that most off-the-shelf models predicting indicators directly obtain low or negative R² (the share of variance explained, where a negative value means worse than predicting the average). Any claim about regional reasoning must first clear this floor in each region separately.
The strongest existing evaluation method is CityLens's own protocol, especially its feature-based regression. In that pipeline, a vision-language model scores 13 visual attributes and a LASSO regression maps those scores to indicators. The file reports that this pipeline consistently outperforms direct prediction and that fine-tuned models exceed R² 0.8 on building height and 0.6 on GDP. However, I found no reported Lacuna result giving a numeric R² for feature-based regression, or a numeric North-versus-South gap. We will therefore reproduce both on our matched cities with the same models. That reproduced gap is the quantity our decomposition must explain. "Beating" this baseline here means attribution rather than accuracy: showing that a statistically significant fraction of the CityLens-style gap disappears under location blocking or under human-verified labels. The same reproduction will not show that fraction disappearing if recall plays no role.
The strongest dynamic-generation competitor is a KBE-DME-style pipeline applied to the same tiles. A vision-language model would generate and answer the multi-hop questions itself from automatically extracted triplets, with no human verification. No reported result found on Lacuna for KBE-DME on geospatial imagery. It will receive the same tiles, the same generation budget and the same depth range as our generator. Our method beats it if KBE-DME's answer keys disagree with human-verified answers more often in sparse regions than in dense ones, and if that disagreement changes its estimated regional gap.
The key ablation is our own pipeline with OpenStreetMap-derived graphs left unverified, in the style of OSMDA. It isolates the human ingredient. Everything else is held fixed, and only the answer key changes. A second ablation removes the location-blocking manipulation and keeps only composition depth. This tests whether composition alone, as Beyond Memorization suggests, already removes the recall effect.
All models are evaluated zero-shot with identical prompts and decoding settings across conditions. The models are Qwen2.5-VL-7B, the backbone of RemoteReasoner, the released checkpoints of Geo-R1 (Unlocking Zero-Shot Geospatial Reasoning via Indirect Rewards) and GeoZero where they are available, and two frontier API models.
Why not just…?
Why not just generate questions automatically from OpenStreetMap and skip the humans? This is the most serious threat, and the study is built to settle it. OSMDA shows that automatic synthesis from OpenStreetMap is cheap and strong on many benchmarks, but its authors report degradation where map coverage is sparse. Not at Home on the Range explains why: rural areas have too few local contributors for the number of features to map. Our ablation measures the rate of human corrections and how much the answer keys flip, separately by region. If corrections are rare and evenly spread, so that the unverified and verified keys give the same gap estimate within confidence intervals, then human verification is unnecessary. We would report that result and recommend the cheaper automated pipeline.
Why not just evaluate on satellite imagery captured after the model's training cutoff? New imagery of an old place does not remove recall of facts about that place, such as its GDP or its tallest district. Beyond Memorization also shows that the presence or absence of post-cutoff decay is an unreliable contamination signal for multi-step questions. We include a post-cutoff split as a baseline condition anyway. If it alone reproduces the recall share that our location-blocking manipulation measures, then the simpler temporal check suffices for this purpose.
Why not just ask the model to geolocate each image and stratify results by whether it recognizes the place? This is a cheap and useful probe, and we run it as a covariate. It measures recognition, though, not whether recognition is used to answer, and a model can benefit from partial recognition it never states. The location-revealed versus location-blocked comparison intervenes on the recall channel directly. We will report how well recognition rate predicts the measured recall share. If the correlation is high, future work can use the probe alone.
Why not just use a larger model and assume the gap closes? CityLens reports that some 12B models beat their 27B siblings on GDP and life expectancy. GeoSQA shows that reasoning fails even when perception is removed entirely. We include models of several sizes so that scale is a measured variable rather than an assumption.
Decisive Checks
The first check asks whether label error is asymmetric. If human annotators change the computed answer for substantially more questions in Global South and rural tiles than in Global North tiles, then fully automated geospatial benchmarks are biased against sparse regions. That would justify the human-in-the-loop component. The second check asks whether recall is asymmetric. If revealing the location raises North accuracy more than South accuracy on recall-open questions, but not on recall-blocked questions, then part of the reported Global North advantage is memorization. The third check is the residual gap in the graph-only condition. A remaining gap there means reasoning itself, not just perception or recall, degrades on unfamiliar scene configurations. That would be the clearest target for training methods such as Geo-R1 and GeoZero.
The hypothesis is refuted if location reveal helps both regions equally and the gap is unchanged on recall-blocked questions. In that case, the CityLens gap is a genuine deficit in perception or reasoning, and that conclusion would itself be worth stating with evidence.
References
- CityLens: Evaluating Large Vision-Language Models for Urban Socioeconomic Sensing
- GeoChain: Multimodal Chain-of-Thought for Geographic Reasoning
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- Not at Home on the Range: Peer Production and the Urban/Rural Divide
- Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Dynamic Multimodal Evaluation via Knowledge-Enhanced Benchmark Evolution
- ToolQA: A Dataset for LLM Question Answering with External Tools
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs
- Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination
- Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models
- RemoteReasoner: Towards Unifying Geospatial Reasoning Workflow
- Unlocking Zero-Shot Geospatial Reasoning via Indirect Rewards
- GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes