Vision-language models, which answer questions about images, score near chance on basic spatial questions such as "what is to the doll's left?" or "which shape results from this fold?". These benchmarks are almost always multiple choice, and the same models are known to favor options by their position in the list and to change answers when a prompt is reworded. So a near-chance score has at least three possible explanations. The model may be guessing. It may be picking by option slot. Or it may be reasoning consistently in the wrong reference frame, for example answering from the camera's viewpoint when asked about a person in the scene.
We propose to tell these apart for each question. We ask the same question many times under controlled, meaning-preserving changes: rotated option orders, paraphrases, and different wordings of the reference frame. Every answer is mapped back into a shared geometric space, such as a direction angle, a count, or a box. We then count how many distinct answer clusters the model produces. The counting uses the eigengap heuristic: build a similarity graph over the answers and find the largest jump in its sorted spectrum, which indicates how many well-separated groups the graph contains. The bet is that the shape of the answer distribution identifies the cause of failure. One tight cluster means a stable answer. Two clusters related by a mirror or 180° rotation mean frame confusion. Clusters that follow option slot mean positional bias. No clear gap means guessing. That shape should also predict errors better than plain disagreement rates.
Core Claim
Spatial failures in vision-language models are structured, not diffuse. Under perturbation, many wrong answers split into a small number of geometrically related hypotheses, most often two that differ by a frame transformation, rather than spreading out at random. A spectral count of answer clusters with graded geometric similarity exposes this structure without access to model internals. The claim has three testable consequences. First, it splits benchmark failures into positional, frame-confusion, and guessing components, which tests whether reported near-chance scores reflect spatial simulation deficits or option-order artifacts. Second, it gives a selective-prediction signal (deciding when the model should abstain) that beats vote share and disagreement on tasks whose answers have geometry. Third, it supports a targeted repair: re-ask with an explicit frame-conversion step only on items diagnosed as frame-confused. If that repair helps those items more than others, the diagnosis is real.
The spectral machinery is justified only where answers are graded, such as directions with angular neighbors, counts, distances, or boxes. On a plain four-option question with unrelated options, counting clusters reduces to counting distinct answers. There we expect parity with simple voting and will report it.
What Prior Work Leaves Open
Several benchmarks motivate the problem.
- Spatial-DISE builds Blender puzzles with near-miss distractors. It reports average accuracy of 28.4% against 25% chance, and attributes 72.5% of failures to reasoning rather than perception.
- ViewSpatial-Bench finds "perspective-blindness" and mid-answer flip-flopping between camera-centric and person-centric frames.
- Vision Language Models See What You Want but not What You See shows no model exceeds roughly 29% chance on PerspectBench, many fall below it because of egocentric bias, and scaling does not help.
None of these papers rotates option order or checks consistency under rewording. Their failure attributions therefore rest on single-presentation answers.
A second group of papers constrains how much we can trust those single-presentation answers.
- The Order Effect shows that shuffling options moves GPT-4o mini by −12.24% on MSMARCO and +10.86% on WebGPT. The direction of the effect is task-dependent.
- Hidden Sensitivity in Spatial Reasoning Evaluation shows that VSI-Bench rankings shift once task-agnostic factors such as ground-truth answer distributions are reweighted away.
Both evidence the confound. The VSI-Bench work corrects at the level of benchmark reweighting, while we diagnose each item from the model's own response pattern.
Three papers supply the methods we build on.
-
A Tutorial on Spectral Clustering gives the eigengap rule. - Efficient Latent Semantic Clustering applies spectral clustering with automatic cluster count to hidden states of sampled text rollouts, in order to deduplicate test-time computation. We differ in three ways. Our setting is black-box. Our similarity is geometric in answer space rather than hidden-state cosine. And our variation comes from controlled input perturbations whose labels we use to interpret the clusters, not from sampling noise.
-
On the Relation between Sensitivity and Accuracy in In-Context Learning introduces SENSEL, which abstains when discrete labels disagree across paraphrases and demonstration orders. It was tested only on text classification. We extend the idea to graded spatial answers and replace a disagreement rate with cluster structure.
- Delving into Spectral Clustering with Vision-Language Representations fuses affinity matrices from several prompts to cluster images, but requires the cluster count to be set by hand. We cluster answers to a single question, not images, and estimate the count automatically.
Research Program
For each benchmark item we build a perturbation ensemble with three axes.
- Option order: $N$ cyclic rotations of the options. This is linear in the number of options, unlike the factorial cost of full permutation.
- Paraphrase: $P$ paraphrases of the question stem.
- Frame wording: $F$ phrasings of the reference frame, for instance "from the person's viewpoint" versus "as the person would see it".
A typical setting is $N=4$, $P=3$, $F=2$, giving 24 queries per item.
Each response is mapped to the content it refers to, independent of the slot where it appeared, and placed in a typed answer space:
- angles on the circle for ViewSpatial's 18 directional labels and for What'sUp prepositions;
- integers or log-scaled values for counts and distances;
- 3D boxes for localization;
- discrete identity for unstructured options.
Similarity between two responses $a_i$ and $a_j$ is a kernel on that space: $W_{ij} = \exp(-d(a_i,a_j)^2/\sigma^2)$. Here $d$ is angular distance, relative error, or one minus box overlap, and $\sigma$ is set from the label geometry, not tuned on test data. With the eigenvalues $\lambda_1 \le \lambda_2 \le \dots$ of the normalized graph Laplacian of $W$, the estimated number of hypotheses is
$$\hat{k} = \arg\max_{k \le k_{\max}} (\lambda_{k+1} - \lambda_k),$$
and the size of that maximal gap, $g = \lambda_{\hat{k}+1} - \lambda_{\hat{k}}$, measures how cleanly the answers separate. A small $g$ means there is no meaningful cluster structure.
Each item is then labeled by comparing its clusters to the known perturbation labels and to the known spatial transformations:
- Stable: $\hat{k}=1$ with a large gap.
- Positional: cluster membership carries high mutual information with the option slot, and low mutual information with paraphrase or frame wording.
- Frame-confused: $\hat{k}=2$ with cluster centers related by left–right mirroring or 180° rotation, at a rate above what the label geometry alone would produce.
- Guessing: small gap.
The confidence score used for abstention combines the share of answers in the dominant cluster with $g$.
The work involves no training. It runs equally on open models, where token probabilities are available as an extra baseline, and on API models.
Baselines
The naive reference point is a single presentation with the benchmark's original option order and prompt, scored by standard accuracy. Confidence for that setting is either a constant (never abstain) or token probability where available. Published numbers on this setting give the anchors:
- Qwen2.5-VL-7B reaches 26.1% on Spatial-DISE, and 47.0% after fine-tuning on Spatial-DISE-12K (Spatial-DISE).
- On ViewSpatial-Bench, GPT-4o reaches 34.98%, InternVL2.5-8B 43.24%, and the fine-tuned MVSM 82.09%.
We reproduce single-presentation accuracy ourselves before comparing.
The strongest competitor for the corrected score is circular evaluation plus majority vote over de-permuted answers, the practice used by CMMU and MathBench in text. It is the obvious way to remove slot preference, and on unstructured multiple choice it captures everything our method does. It is run on exactly the same 24 responses per item, so compute is identical. No reported result found on Lacuna for circular evaluation on Spatial-DISE, ViewSpatial-Bench, or PerspectBench.
The strongest competitor for abstention is SENSEL from On the Relation between Sensitivity and Accuracy in In-Context Learning, computed on the same perturbation ensemble. That paper reports a +4.1 point gain in area under the F1-coverage curve over MaxProb on GPT-J-6B for text classification. No reported result found on Lacuna for SENSEL on any spatial or multimodal benchmark. We also run two cheaper geometric baselines that pose the most direct threat:
- Mean pairwise affinity: the average of $W_{ij}$, which measures dispersion with no spectral analysis.
- Vote share of the modal answer.
All methods see the same responses. All hyperparameters, including $\sigma$, $k_{\max}$, and the weighting between cluster share and gap, are fixed on a held-out 10% split of each benchmark.
Two ablations isolate the key ingredients:
- Graded versus exact-match similarity. Replacing graded similarity with exact-match similarity tests whether answer geometry matters.
- Stratified versus reshuffled perturbation labels. Randomly reshuffling the perturbation labels before attribution tests whether the positional and frame diagnoses come from the controlled design rather than chance clustering.
The proposal beats the baselines if three things hold on graded-answer benchmarks:
- Its abstention curve dominates mean pairwise affinity and SENSEL with non-overlapping bootstrap intervals.
- Its frame-confusion label predicts which items the targeted repair fixes.
- Its corrected accuracy matches or exceeds circular majority vote.
Why not just…?
Why not just run circular evaluation and majority vote? For four-option questions with unrelated distractors, as in much of Spatial-DISE, this is a real threat and we expect it to match our corrected accuracy. We will report parity there rather than hide it. What voting cannot say is why answers disagree. It also fragments graded answers: "front-left" and "left" count as different votes on ViewSpatial's 18 labels, so a frame flip and random scatter look the same. The frame-repair experiment below tests whether that distinction has practical value.
Why not just use mean pairwise disagreement or SENSEL as the confidence score? Dispersion scores cannot separate two tight, competing hypotheses from diffuse noise, because both lower the mean affinity. The perspective literature predicts the bimodal case specifically: flip-flopping in ViewSpatial-Bench and systematic egocentric answers in PerspectBench. If bimodality turns out rare, mean affinity will tie our score, and we will conclude the spectral step is unnecessary for abstention.
Why not just use token probability or entropy? This needs access to logits, which rules out API models. On the Relation between Sensitivity and Accuracy in In-Context Learning also reports that perturbation-based signals beat both on text, because biased models are confidently wrong. A model that consistently answers in the camera frame should likewise be confident and wrong, and we test this directly on open models.
Why not just fine-tune on spatial data? Fine-tuning does raise scores, with MVSM at 82.09% and Spatial-DISE-12K at 47.0%. However, What's "up" warns that it can instill dataset shortcuts. Fine-tuning is complementary to our diagnosis, not a substitute for it, so we apply the diagnosis to fine-tuned models too. If they show only stable or guessing patterns, the frame-confusion component has been trained away. If they show positional clusters, their gains are partly artifact.
Decisive Experiments
The study covers Spatial-DISE, ViewSpatial-Bench, PerspectBench, What'sUp, and 3DSRBench. The models are Qwen2.5-VL-7B, InternVL2.5-8B, MVSM, SpatialThinker-7B if weights are available, and GPT-4o.
The first check quantifies how much of near-chance performance is positional. It compares single-presentation accuracy, circular-consistent accuracy, and the fraction of items labeled positional, per benchmark and model. It also asks whether Spatial-DISE's 72.5% reasoning attribution survives once slot preference is removed.
The second check tests the structure hypothesis. On ViewSpatial-Bench and PerspectBench, the rate at which two-cluster items are related by mirror or 180° rotation must exceed a null rate. That null rate comes from drawing cluster-center pairs according to the benchmark's label geometry. If the rate does not exceed the null, the core claim fails.
The third check is the causal test. Items labeled frame-confused are re-asked with an instruction to first state the person's facing direction and then convert. A matched random sample of other wrong items gets the same instruction. The claim requires a larger accuracy gain on the diagnosed items than on the controls.
The fourth check is selective prediction. It reports area under the F1-coverage curve and risk at fixed coverage against all baselines, under equal query budgets. A budget sweep from 6 to 24 queries shows where the spectral signal stops adding information over cheaper dispersion scores.
References
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Vision Language Models See What You Want but not What You See
- The Order Effect: Investigating Prompt Sensitivity in Closed-Source LLMs
- Hidden Sensitivity in Spatial Reasoning Evaluation: Diagnosis and Re-ranking with VSI-Bench
- A tutorial on spectral clustering
- Efficient Latent Semantic Clustering for Scaling Test-Time Computation of LLMs
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Delving into Spectral Clustering with Vision-Language Representations
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning