Do Spatial Benchmarks Measure Space or Familiarity?

Vision-language models, systems that turn an image into tokens a language model can read and then answer questions about it, are known to be weak at simple spatial questions such as "is the cat under the table?". The benchmarks that report this weakness may still overstate what models can do, for two reasons. First, many natural-image spatial questions can be answered from object identity alone: cups are usually on tables, so guessing "on" is right most of the time. Second, the images and captions behind those questions often come from datasets the models were trained on.

We propose to split every spatial-benchmark score into three parts. The first is what the model gets from the scene description (object names) alone. The second is what it gets from having seen these particular images before. The third is what it actually reads from the pixels. We then locate where in the model the true positional signal is lost. The candidates are the vision encoder, the small network that translates visual features into the language model's input space (the "projector"), and the language model's readout. The final step is to test whether a projector-level fix closes the remaining gap.

The practical stakes are simple. If current gains on spatial benchmarks come mostly from learning which object pairs usually appear in which configurations, then the field is improving a lookup table, not spatial perception. Methods that report spatial improvements, including new projectors, synthetic spatial pretraining, and reinforcement learning with spatial rewards, would then need to be re-scored on items where the familiar arrangement is false.

Core Claim

Our central claim is that a large and unmeasured share of natural-image spatial accuracy comes from object-pair co-occurrence priors and from overlap with training images. On items whose true relation contradicts the prior, current models fall to or below chance. We also predict that the positional information needed to answer these items is still linearly decodable at the projector output, and that it is overridden later by the language model's prior.

If that localization holds, the right intervention is not a better projector. It is a change in how the language model weighs projected spatial evidence against its prior. If instead the signal is already gone at the projector output, locality-preserving projectors are the right lever, and we would expect their reported spatial gains to concentrate on prior-violating items. The study is designed so that either outcome is informative.

Grounding in Prior Work

Evidence that motivates the claim. What's "up" with vision-language models? holds object identity fixed and moves objects around. It found that models default to an "on" prior, that spatial prepositions appear in under 0.2% of web captions, and that the best model reached 56% against 99% for humans. That study controls the prior by design but cannot say how much of the scores on natural-image benchmarks the prior explains.

Visual Spatial Reasoning (VSR) is built from MS COCO images. It reports a plateau near 70% against 95.4% for humans, with sharp drops on its concept zero-shot split. That drop is indirect evidence of object-specific shortcuts, and the COCO source makes contamination plausible, because COCO images are common in instruction tuning.

Where spatial information might be lost. SpaRRTa shows that spatial information lives mainly in patch-level features and is lost under global pooling. That points to the interface between vision and language as a suspect, but SpaRRTa does not probe a multimodal language model's projector.

Methods we reuse as interventions. Three projector designs claim better spatial behaviour:

  • Honeybee adds convolutional and deformable-attention abstractors.
  • SAEP aggregates multiple encoder layers on a 2D grid and reports a spatial score rising from 46.7 to 51.4.
  • Deep Pre-Alignment refines visual tokens with a small language model before they reach the main one.

None of these papers separates prior-consistent from prior-violating items. Vision Language Models Map Logos to Text via Semantic Entanglement in the Visual Projector shows that projectors can map visual input onto text-token meanings. That is a mechanism by which "what" could crowd out "where".

Closest work on co-occurrence bias. PURGE partitions data into shortcut-prone and object-grounded subsets and unlearns object-background correlations. Visual Contrastive Decoding penalizes outputs driven by statistical expectation rather than visual evidence. Both target whether an object is present, not how two objects are arranged. We adapt both as baselines.

Methods for fresh and controlled evaluation. For contamination-resistant data we borrow two methods. MMBench-Live builds fresh multimodal items through real-time search and verification. Procedurally generated items follow the approach of A Cognitive Battery for Foundation Models, including its controlled distractor injection.

A constraint from the contamination literature. The LLM contamination literature, such as Can Large Language Models Reason? A Characterization via 3-SAT and Unveiling Causal Reasoning in Large Language Models, shows collapse once items leave the training pattern. This constrains our design: any "improved" method must be scored on items it could not have seen.

What the search did not show. The research file I worked from was truncated before its section on existing intersections. I therefore cannot rule out an existing paper that does this exact decomposition. Among the papers visible to me, none splits spatial accuracy along both prior and contamination axes, and none probes the projector for relational information.

Research Program

Step 1: a prior score for every item. For each spatial item with objects $a$ and $b$ and asserted relation $r$, we define a prior score $\pi(r \mid a,b)$. It averages two estimates:

  • the probability a text-only language model assigns to $r$ given only the object names, and
  • the relative frequency of $r$ between $a$ and $b$ in a large caption corpus, after parsing relations.

Items with high $\pi$ for the true relation are prior-consistent. Items where the true relation has low $\pi$ are prior-violating. The headline quantity is the prior gap:

$$\Delta_{\text{prior}} = \text{Acc}(\pi > \tau) - \text{Acc}(\pi < 1-\tau),$$

Here Acc is model accuracy on the subset and $\tau$ is a threshold, fixed in advance (e.g. 0.7) and varied in a sensitivity analysis. For a model that reads relations from the image, $\Delta_{\text{prior}}$ should be near zero.

Step 2: a freshness gap. For VSR items we collect new photographs of the same object pair in the same relation. We build them with an MMBench-Live-style search-and-verify pipeline and have humans confirm the relation. The difference in accuracy between original COCO images and these fresh images is the freshness gap. It isolates image-level memorization from the pair-level prior.

Step 3: rendered counterfactual pairs. We render matched scenes in the style of SpaRRTa's Unreal Engine setting, in which the same object pair appears in both the typical and the inverted relation, for example a cup on a table and a cup under a table. Because these scenes are generated fresh, they cannot be contaminated, and they hold appearance fixed.

Step 4: locating the loss. We fit linear probes for the relation at three points:

  • vision-encoder patch features pooled over each object's region,
  • projector outputs at the same positions,
  • language-model hidden states at the answer position, across layers.

Probes are trained on prior-violating items only, so they cannot succeed by recovering object identity. The layer at which probe accuracy drops identifies where the evidence is lost. If probes succeed but the answer is wrong, the evidence was present but overridden.

Step 5: a targeted intervention. Depending on what Step 4 finds, we train one of two fixes. If information is lost in the projector, we fine-tune the projector alone, with the language model frozen, on counterfactual pairs, using an auxiliary loss that keeps relational probe accuracy high at its output. If information is lost in the language model, we train with the projector frozen on the same pairs, and add a small amount of contrastive decoding against an object-names-only input.

Baselines

Naive reference points. The do-nothing reference is the base model. It is a LLaVA-style model with a plain multilayer perceptron projector, evaluated zero-shot on every split. Alongside it we report two trivial predictors: a blind text-only answer given the object names, and the majority relation for each object pair. Any model whose prior-violating accuracy does not exceed these predictors is not using the image for space.

Strongest projector baselines. These are SAEP and Honeybee's C-Abstractor, retrained on the same instruction data and language model as our base, with matched token counts (144) and matched training steps. SAEP reports a spatial-task increase from 46.7 to 51.4 SAEP, and Honeybee-13B reports 1950.0 on MME Honeybee. No reported result was found on Lacuna for either method on a prior-violating spatial split. We will run both ourselves. A projector "beats" our intervention only if it raises prior-violating accuracy, not just the overall average.

Strongest data baseline. This is synthetic spatial pretraining in the style of SpatialVLM, which reports 75.2% on binary spatial predicates. We fine-tune our base on a SpatialVLM-style dataset with the same number of gradient steps and examples as our counterfactual set. We also evaluate SpatialThinker, a reinforcement-learning method with spatial rewards that reports +12.1% over GPT-4o on 3DSRBench, using its released recipe and our base. Our method beats these baselines if it achieves higher prior-violating accuracy and a smaller $\Delta_{\text{prior}}$ at equal or better prior-consistent accuracy.

Inference-time baseline. Visual contrastive decoding is applied unchanged, plus a variant that contrasts against a blank image with object-name text. Both are tuned on the same held-out validation set as our method.

Ablation of the key ingredient. We run the same counterfactual data with randomly re-paired relations, which keeps data volume and object diversity but removes the prior-violating structure. We also run the same data with the update applied to the other component, the language model instead of the projector or vice versa. This tells us whether the prior-violating structure, and the location of the fix, are what matter.

Why not just…?

Why not just use What'sUp, which already controls for object identity? What'sUp establishes that the prior exists, but on 820 tripod images of a few objects. It cannot say how much of the roughly 70% plateau on natural VSR images the prior explains, and it says nothing about contamination. Our prior score applies to any existing benchmark. We will report What'sUp results as an anchor and check that our score assigns its "on"-default failures low $\pi$.

Why not just run a blind text-only baseline? This is the most serious competitor, and we include it. A blind baseline measures how much the prior could explain. It does not show whether the full model actually uses the prior, separate pair-level priors from image-level memorization, or locate where visual evidence is dropped. The decisive comparison is per item. If the full model's errors on prior-violating items match the blind model's answers far more often than chance, the prior is being used, not just available.

Why not just fine-tune on more synthetic spatial data? SpatialVLM and SpatialThinker do raise average scores. However, What'sUp reports that fine-tuning on existing spatial datasets failed to generalize and degraded performance on controlled images. Synthetic data may simply teach new, broader priors. Our prior-violating and rendered counterfactual splits measure this directly. If synthetic fine-tuning closes $\Delta_{\text{prior}}$ as well as our targeted intervention does, we will report that the targeted fix is unnecessary.

Why not just swap in a locality-preserving projector? If Step 4 shows that relational information is already lost at the projector output, this is indeed the answer, and our intervention reduces to choosing the best such projector. The probe results settle the question. If plain projector outputs already support high relational probe accuracy, better projectors cannot be the main lever, and their reported gains should appear mostly on prior-consistent items.

Proposed Experiments and Decisive Checks

Benchmarks and splits. We evaluate on VSR and What'sUp, split by prior score. We add fresh VSR re-photographs and roughly 5,000 rendered counterfactual pairs covering at least the What'sUp prepositions and VSR's orientation relations, where VSR reports its worst failures. As a check outside the image domain, we also evaluate the CausalSpatial occlusion subset CausalSpatial.

Models. The open models are our base, Honeybee, SAEP and Deep Pre-Alignment variants at roughly 7B parameters. The blind and majority predictors are reference points.

Metrics. We report accuracy on each split, $\Delta_{\text{prior}}$, the freshness gap, the rate at which errors agree with the blind predictor, and relational probe accuracy at each of the three probe points.

Decisive check for the measurement claim. This claim is refuted if $\Delta_{\text{prior}}$ is small, under 5 points, for the base model on VSR, and if the freshness gap is also small.

Decisive check for the localization claim. This claim is the comparison between probe accuracy at the projector output and accuracy of the answers on rendered counterfactuals. A large surplus of probe accuracy over answer accuracy means the evidence survives projection and is overridden downstream.

Decisive check for the intervention. The fix must raise rendered counterfactual accuracy above both the SpatialVLM-style fine-tune and the best projector baseline, at equal compute. It must not reduce prior-consistent accuracy by more than 2 points. It must also hold on fresh re-photographs that none of the compared systems could have seen during training.