Is the Map in the Pixels or in the Memory? A Renewable Audit of Zero-Shot Geospatial Relational Reasoning

Multimodal foundation models are increasingly asked questions about satellite images that involve relations between things: is this road connected to that settlement, does this field touch the river, did the flood reach the bridge. When a model answers correctly with no task-specific training, we call it zero-shot reasoning. There are two cheaper explanations. The model may remember the place, because maps, articles, and public benchmark question-answer pairs about well-documented cities sit in its training data. Or the benchmark may be easy for a structural reason: it asks only about relations that exist, so "yes" is nearly always right, even though most pairs of objects in a real scene are unrelated. This project builds an evaluation that separates three things: perception, place memory, and the base-rate shortcut. It then measures how much each one inflates reported zero-shot performance, and whether the inflation is larger in the well-mapped Global North than in the Global South.

The core device is a natural experiment that the Earth runs for free. Satellites revisit every location every few days, so we can ask the same relational question about the same place in an image taken before a model's training cutoff and in one taken after. Some places change after the cutoff: new roads get built and floods inundate new ground. On those tiles, the correct answer now contradicts whatever the model could have memorized. A model that reads the image follows the new answer. A model that recalls the place gives the old one.

Core Claim

We claim that current zero-shot geospatial relational benchmarks overstate visual relational ability through two separable channels, and that both channels can be measured directly.

The first channel is memorization of places and benchmark items. We measure it by how often a model agrees with the pre-change map on tiles whose relation flipped after its cutoff. No lexical or perplexity contamination detector is involved, and those detectors are known to break under simple paraphrase (On The Fragility of Benchmark Contamination Detection in Reasoning Models; Rethinking Benchmark and Contamination for Language Models with Rephrased Samples).

The second channel is relational sparsity. In a scene with $N$ entities there are $N(N-1)$ ordered candidate pairs, and most of them hold no relation; in text, over 97% of DocRED entity pairs are unlinked (DocRED). Question-answer benchmarks built by writing questions about annotated relations hide this, because the question itself presupposes that the relation exists. We measure the second channel as the gap between accuracy on positive-only questions and precision and recall over the full candidate-pair space. Within that space, negatives are stratified by spatial distance. Following Tobler's law, near unrelated pairs are the hard negatives, while far pairs are trivially unrelated (Mission Critical – Satellite Data is a Distinct Modality in Machine Learning).

What is new is this factorial design. It crosses three factors: before versus after the cutoff, seen versus held-out region, and positive-only versus full-pair sampling. It also adds changed-relation items that act as counterfactuals reality has already produced. Existing work varies at most one of these factors.

Relation to Prior Work

Three strands motivate the proposal but leave its question open.

EarthVL shows that Earth vision lacks object-relational reasoning. It builds EarthVLSet, with 10.9k sub-meter images from 17 countries and 761.5k question-answer pairs, and trains EarthVLNet, which grounds a language model in segmentation masks. Its benchmark is static, publicly hosted, and question-driven. We treat it as the clearest example of the setting we audit, not as a competitor to beat on its own terms.

PANGAEA and Fine-tune Smarter, Not Harder evaluate geospatial foundation models, meaning large image encoders pre-trained on satellite data, across continents and on geographic hold-out sets. Both are limited to pixel segmentation and classification, use fine-tuning or frozen probing rather than true zero-shot evaluation, and do not audit contamination. We reuse their hold-out logic and their strongest adapted models as a non-language-model pipeline.

On the contamination side, DyVal generates fresh reasoning problems from random graphs and finds that models strong on static benchmarks score near zero on fresh instances. LiveBench refreshes questions from post-cutoff data. DyVal's generator cannot produce realistic imagery. Post-cutoff refresh alone cannot distinguish reading an image from remembering a place whose layout has not changed. Our changed-relation items are designed to close exactly that gap.

From the relational sparsity literature we borrow the evaluation lens rather than a model. DocuNet and EP-RSR show that scoring over the full candidate-pair matrix exposes failures that per-question scoring hides. EP-RSR's pair-retrieval pipeline raises DWIE F1 from 56.38 to 74.38 over document-wide extraction.

One line of work constrains the design. Model Evaluation for Geospatial Problems and Evaluation Challenges for Geospatial ML warn that pushing test regions far from training regions adds covariate shift, which makes estimates pessimistic. We therefore use buffered blocks and also report importance-weighted estimates, so that a geographic gap is not simply an artifact of distance.

Research Program

Data sources. The audit pairs freely available, frequently revisited imagery with openly licensed vector map geometry, which supplies checkable ground truth.

Two resolution tiers. The 10 m tier uses Sentinel-1 and Sentinel-2 acquisitions. Its relations are visible at that scale: field–river adjacency, settlement–road connectivity through the road graph, and inundation of road segments during floods. The flood relations follow the labeling practice and global event coverage of Sen1Floods11. The sub-meter tier uses EarthVLSet's 17 countries as the "seen, pre-cutoff" reference arm. It is restricted to geometric relations that we can re-derive from vectors: containment, adjacency, and nearest-neighbour ordering.

Candidate pairs and item types. For every tile, the generator lists all entity pairs and labels each pair for each relation type from the map geometry. It then samples evaluation items in two ways. Positive-only sampling mirrors existing benchmarks. Full-pair sampling preserves the true base rate and bins negatives by centroid distance. Changed-relation items are mined by differencing pre- and post-cutoff vector snapshots and confirming each flip with change detection on the imagery; a human verifies a subset.

Prompt variants. Each item is rendered in several phrasings and option orders, because zero-shot accuracy can shift with label position alone (ABCD: All Biases Come Disguised).

Regions. Regions are split into buffered spatial blocks, labeled Global North or Global South, and assigned to seen or held-out arms relative to EarthVLSet's coverage.

Outputs. The program produces three numbers per model per region. The memory index is the excess agreement with the pre-change answer on changed items, beyond what the model's error rate on unchanged post-cutoff items predicts. The sparsity gap is positive-only accuracy minus full-pair F1, with full-pair F1 also reported per distance bin. The geographic gap is held-out-region minus seen-region full-pair F1 on post-cutoff, unchanged tiles. Because ground truth comes from geometry and the imagery stream is continuous, the benchmark can be regenerated every few months. Releasing the generator rather than a fixed item set limits future leakage.

Baselines

Naive reference points. The do-nothing reference is the constant predictor that answers "no relation" for every pair. Under full-pair sampling it scores high accuracy and zero recall, which is exactly why accuracy is not our headline metric. Beside it we run a distance-only prior that predicts adjacency or connectivity whenever two entities are nearer than a threshold fit on seen regions. Any model that fails to beat this prior on near-distance negatives is not using visual content.

Image-blind baseline. This is the most important ablation. The same multimodal model receives the tile's coordinates and place name but no pixels. Its accuracy is a direct estimate of how much can be answered from place memory alone. We expect it to be well above chance in well-mapped regions. If it is not, the memory channel is small.

Strongest relational vision-language system. This is EarthVLNet, which reports 81.06% multiple-choice VQA accuracy on EarthVLSet and is reported to significantly outperform GPT-4o. We run the released model zero-shot on our items. On the sub-meter tier we also run GPT-4o-class general models under identical prompts, option permutations, and decoding budgets. Beating it means higher full-pair F1 on held-out, post-cutoff tiles, plus a memory index statistically indistinguishable from zero.

Strongest non-language-model competitor. The pipeline segments the image with Prithvi 2.0 300M adapted by LoRA and then computes relations geometrically from the predicted masks. Fine-tune Smarter, Not Harder reports 87.57% mIoU for this model on the Sen1Floods11 geographic hold-out set, against 82.07% for full fine-tuning. The pipeline is adapted only on seen regions with a fixed tuning budget and is never shown target tiles. I found no reported result on Lacuna for zero-shot relational extraction by any geospatial foundation model, so its relational scores will be new.

Why not just…?

Why not just use post-cutoff imagery, as LiveBench does for text? We do use it, but by itself it cannot separate reading from remembering. A road that existed before the cutoff still exists in a post-cutoff image, so a model that knows the town answers correctly without looking. LiveBench works for text because new questions carry new content; a re-imaged place carries old content. The changed-relation items, together with the image-blind baseline, are what isolate memory. If the memory index turns out near zero for all models, this alternative was sufficient, and we will say so.

Why not just run spatial cross-validation or a geographic hold-out set? Hold-out regions address shift but not contamination, because a held-out region for our training split can still be well documented on the web. Distance-based splits also bias estimates pessimistically by adding covariate shift (Model Evaluation for Geospatial Problems). We keep buffered blocks as one axis of the design and report importance-weighted estimates alongside raw ones. Hold-outs are one factor in the design, not a replacement for it.

Why not just run a contamination detector on EarthVLSet? Lexical-overlap and perplexity detectors fail under paraphrase (Rethinking Benchmark and Contamination for Language Models with Rephrased Samples). They also need access to likelihoods or training data that closed multimodal models do not expose, and they detect item leakage, not place knowledge. Our behavioural test needs only the model's answers.

Why not just balance yes/no questions to 50/50? Balancing removes the "always yes" shortcut but evaluates at the wrong base rate. It also usually draws far, easy negatives. Distance-stratified full-pair scoring tests whether a model rejects plausible but absent relations, which is where the text-extraction literature finds models fail (EP-RSR). We report the 50/50 number too, so the reader can see how much it hides.

Why not just use segmentation plus geometry and skip language models? This is a real threat. For purely geometric relations, the LoRA-Prithvi pipeline may win outright, and PANGAEA already found strong supervised UNets winning on 6 of 11 datasets. If it does, that is a finding about what "relational reasoning" claims for vision-language models currently amount to. To give language models room to show something the pipeline cannot, we include functional relations, such as whether a settlement is reachable by road during a flood, that require composing several geometric facts.

Proposed Experiments

The first experiment establishes the sparsity gap. On seen, pre-cutoff EarthVLSet tiles, we compare positive-only accuracy with full-pair F1 for every model. Following DocRED-scale sparsity, we predict a drop of tens of points, concentrated in the nearest distance bin.

The second experiment is the decisive memory check. We collect at least several hundred verified changed-relation items per region class and compute each model's memory index with bootstrap intervals, using the image-blind baseline as a reference. The hypothesis fails if the index is near zero everywhere. It is supported if the index is positive and larger in Global North blocks.

The third experiment estimates the geographic gap on post-cutoff, unchanged tiles. It compares seen and held-out blocks, with raw and importance-weighted estimates and with option-order permutations, so that gaps are not driven by prompt sensitivity.

The fourth experiment compares the segmentation-plus-geometry pipeline against vision-language models on geometric versus functional relations. Its result tells us whether language-model reasoning adds anything once the base rate and memory are controlled.

All results are reported as full-pair F1 per distance bin, memory index, and geographic gap. The generator, vector-derived labels, and imagery identifiers are released so that later model releases can be audited on fresh tiles.