Do Spatial Demonstrations Reach the Answer? Probing and Repairing In-Context Spatial Reasoning in Vision-Language Models

Vision-language models, meaning language models that also read images, fail at simple spatial questions such as whether a mug is to the left or the right of a plate. Showing them a few worked examples in the prompt, which is called in-context learning, barely helps. Separately, probing studies train small classifiers on a model's intermediate activations, and they find that the correct spatial relation is often still readable in the middle layers. It is lost somewhere before the model writes its answer. This proposal connects those two observations. We follow what a few spatial demonstrations do inside the model, layer by layer, to find out whether in-context learning fails because the demonstrations never reach the spatial information, or because they reach it and the final layers override it.

The diagnosis then suggests a cheap repair. Spatial relations have a property that factual truth lacks: flipping an image horizontally swaps left and right, and flipping it vertically swaps above and below. That gives a free consistency check. The few demonstrations fix which direction in activation space means "left". The flip constraint then lets us refine that direction on unlabeled images, much as Contrast-Consistent Search refines a truth direction using negations. The refined direction can be used to read the answer directly from the middle layers or to correct the final layers. The point of the work is to learn where demonstrations fail, and then to test whether a label-light repair built on that diagnosis beats existing training-free fixes.

Core Claim

We hypothesize that few-shot spatial demonstrations mostly tell the model what kind of question is being asked, but do not make the image-specific relation more readable at the query position. The relation information is already present in intermediate layers in zero-shot prompts, and the late layers suppress it in favor of language habits such as answering "on" for mugs and tables. If this is right, three things should hold. Compressing demonstrations into a single "task vector" (an activation summarizing the task, patched into a fresh forward pass) will help spatial questions much less than it helps semantic ones. A readout fitted to the demonstrations' own middle-layer activations will beat putting those same demonstrations in the prompt. And flip-consistency will let that readout work with very few labels while relying less on object-identity shortcuts.

This differs from the closest work as follows. Multimodal Task Vectors compresses many demonstrations into activations patched into selected attention heads of Qwen-VL. It was tested only on semantic classification and document question answering. We use it as an instrument, and we predict that it partly fails on spatial tasks. GRASP diagnoses the same mismatch between middle and final layers and fixes it per image by searching over positional encodings, with intervention strength and layer tuned by hand. We instead let demonstrations choose the layer and direction, and we ask what in-context learning itself is doing, a question GRASP leaves open. Grounding Isn't Knowing finds that object coordinates are decodable around layers 12–14 of LLaVA. It also shows that localization heads and relation heads are largely separate. It does not ask how demonstrations change either set of heads.

Research Program

The first stage is measurement. We use LLaVA-1.5 (7B and 13B), Qwen-VL-7B (to match the Multimodal Task Vectors setup), and Qwen2.5-VL. On WhatsUp and VSR questions, we record activations at the final query token in every layer for prompts with 0, 1, 4, and 8 demonstrations. At each layer we fit linear probes, trained on a held-out probe set, to decode the gold relation. In parallel we use a logit-lens readout to measure how much the relation shows up in the vocabulary distribution.

Three curves come out of this. The first is relation decodability by layer as the number of shots changes. The second is the gap between peak middle-layer decodability and final-layer output accuracy. The third is a causal check: we patch activations from the flipped image into the original pass, and record at which layers the patch changes the answer. This applies the counterfactual patching of Grounding Isn't Knowing, but uses flips instead of object removal. The flip is a cleaner counterfactual because it leaves object identity untouched.

The reading of these curves determines the diagnosis. If shots raise middle-layer decodability, demonstrations are doing useful work. If shots raise decodability but not output accuracy, the late layers are suppressing the signal. If shots raise neither, demonstrations only set the task format. This is also how we test the claim in Reasoning After Perception that visual information is destroyed before reasoning begins. That claim predicts low decodability at every layer. GRASP's findings predict high middle-layer decodability. Both may be right on different benchmarks, and middle-layer decodability is the variable that should separate them.

The second stage is the repair, called flip-consistent demonstration readout. At a chosen layer $\ell$, let $h(x)$ be the query-token activation for image–question input $x$. Let $\tilde{x}$ be the same question with the image flipped along the axis that swaps the relation. We fit a direction $w$ and bias $b$ by minimizing, over unlabeled images,

$$ \big[\sigma(w^\top h(x)+b) + \sigma(w^\top h(\tilde{x})+b) - 1\big]^2 + \min\big(\sigma(w^\top h(x)+b),\, \sigma(w^\top h(\tilde{x})+b)\big)^2, $$

where $\sigma$ is the logistic function. The first term says that $x$ and its flip must receive opposite relation scores. The second term prevents the trivial solution of 0.5 everywhere. This mirrors the negation-consistency objective of Discovering Latent Knowledge, with flipping in place of negation. The $k$ labeled demonstrations fix the sign of $w$ and pick the layer $\ell$ by held-out agreement. That replaces the manual layer choice GRASP needs.

We then use $w$ in two ways. In read mode, the answer is taken directly from the probe score. In write mode, we add $\alpha\, w$ at layer $\ell$ and at the late layers where GRASP finds suppression. The strength $\alpha$ is chosen as the smallest value that makes the model's own text answers agree with the probe on the demonstrations.

One caveat governs scope. Flipping works cleanly only for left/right and above/below. Front/behind and depth relations have no free flip, and neither do VSR's object-centric relations such as "facing", which Visual Spatial Reasoning flags as a hard reference-frame case. For those relations we fall back to the purely supervised $k$-shot probe and report results separately. We do not claim the method covers them.

Baselines

The naive reference points are the zero-shot model and the same model given $k$ demonstrations in the prompt. Both run with identical demonstrations, image resolution, and decoding. Kamath et al. report the best model on WhatsUp at 56% accuracy, with zero-shot contrastive models often near random. Humans reach 99%. A method has to clearly beat the few-shot prompt with the same $k$ to justify white-box access at all.

The strongest existing method is GRASP. It reports LLaVA-1.5 rising from 60.3% to 86.4% accuracy on WhatsUp, a gain of up to 26.1 points. We found no result on Lacuna for GRASP on VSR. We will rerun GRASP on the same model checkpoints, the same WhatsUp subsets (Controlled_A, Controlled_B, COCO, and Visual Genome), and VSR. GRASP gets a hyperparameter search over $\alpha$ and layer equal in size to ours, using the same held-out tuning images. Beating it means higher accuracy at equal or lower latency, without per-image counterfactual search. It also means a smaller drop on out-of-distribution splits: VSR's concept zero-shot split, and transfer from the controlled WhatsUp subsets to the COCO subset. Matching GRASP's in-distribution accuracy while using only $k \le 8$ labels would also count as a result, because GRASP's hyperparameters are tuned by hand.

The in-context-learning competitor is Multimodal Task Vectors, run on Qwen-VL-7B with its REINFORCE head selection. It uses the same pool of labeled demonstrations, up to the many-shot regime that paper uses. That paper reports 45.6% on VizWiz against 44.3% for 8-shot prompting. We found no result on Lacuna for it on any spatial benchmark. A third baseline is a contrastive mean-difference steering vector built from left versus right demonstrations, which captures the spirit of Dynamic Multimodal Activation Steering without its object detector.

Two ablations isolate the key ingredient. The first is the purely supervised $k$-shot linear probe, which drops the flip term. The second is the flip-consistent readout with random sign, which drops the labels. Together they show what the consistency term adds over plain few-shot probing.

Why not just…?

Why not just add more demonstrations to the prompt? Kamath et al. report near-random few-shot behavior on WhatsUp, and Reasoning After Perception finds 3-shot prompting gives negligible gains on its perception test. Neither explains why. Our measurement stage answers this directly: if extra shots do not increase middle-layer decodability, more shots cannot help. Long multimodal prompts are also costly; Multimodal Task Vectors reports 20.6 GB of GPU memory for 16-shot prompting in Qwen-VL.

Why not just train a linear probe on labeled spatial data and read the answer out? This is the real threat. It is why the supervised $k$-shot probe is a baseline and not a strawman. If a supervised probe trained on a few hundred labels matches our method, the repair adds only label efficiency, and the main contribution becomes the diagnosis. We will report accuracy against the number of labels for both. The flip-consistent version must win clearly at $k \le 8$ and on the concept zero-shot split, where supervised spatial models are known to collapse onto object shortcuts Visual Spatial Reasoning. If it does not, we will say so.

Why not just fine-tune on spatial data? Kamath et al. report that spatial fine-tuning lowered performance on the controlled WhatsUp images. VSR reports sharp drops on the concept zero-shot split. We include a LoRA fine-tune on the same label budget as a reference. We expect it to win in distribution and lose on transfer, and that is the comparison that matters.

Why not just use GRASP? GRASP already fixes much of WhatsUp, so our method must earn its place on transfer, label cost, and automatic layer and strength selection. It must also say something GRASP cannot: whether demonstrations themselves carry usable spatial signal.

Proposed Experiments

The first decisive check is the shot-by-layer decodability analysis. If eight demonstrations raise final-layer accuracy without changing middle-layer decodability, or if decodability is already high zero-shot, the core claim of late suppression holds. If decodability is near chance at every layer for some benchmark, that benchmark sits in the regime of perceptual information loss, and we predict no activation method will help there. As a contrast case we run the probes on the Turing Eye Test of Reasoning After Perception, where we expect exactly that outcome.

The second check compares the gain from Multimodal Task Vectors over prompting on WhatsUp and VSR against the gain the same method gets on a semantic task, VizWiz, under the same protocol. A much smaller spatial gain supports the claim that task vectors encode the format of the question and not the relation.

The third check is the main accuracy comparison against GRASP, the prompting baselines, and the ablations. It is run on all WhatsUp subsets and both VSR splits, with cross-subset transfer.

The fourth check audits locality. It is motivated by the fragile locality that task-vector negation audits found in robot policies. After write-mode steering on left/right, we measure changes in above/below accuracy, front/behind accuracy, and object-identity questions on the same images. A useful repair should leave those changes within noise.

A continuous extension to TangramSR would replace its external geometry verifier with regression probes on rotation, where rotating the image yields a known shift in the target. We treat this only as a follow-up once the discrete results are in.