Symmetry-Pooled Uncertainty for Multi-Hop Spatial Reasoning and Its Judges

Language and vision-language models fail at simple spatial questions such as which object is to the left, or where X lies relative to Z given a chain of pairwise relations. Worse, they often fail confidently. Ask the same question ten times and the model may give the same wrong answer ten times. The standard tool for spotting unreliable answers is semantic entropy: sample several answers, group the ones that mean the same thing, and measure how spread out the groups are. It cannot see this failure, because a consistent error has zero spread. Spatial tasks have something that factoid trivia lacks, though: exact symmetries with known effects on the answer. Mirror the image and "left" must become "right". Rotate every compass relation in a text story by 90° and the answer must rotate with it. Rename the entities and the answer must not change. We propose to sample answers under these transformed inputs, map each answer back through the known transformation, and pool everything before computing entropy. A model that says "left" both before and after mirroring is consistent within each view but contradicts itself once the views are pooled, and the pooled entropy shows it.

The opportunity is that one cheap, black-box signal can serve two consumers that currently lack calibrated uncertainty on spatial tasks. The first is the reasoner itself, across multi-hop chains, where we can score every intermediate hop and find the first one that broke. The second is the model acting as an automated grader (an "LLM judge") of those chains, whose verdict on a spatial step is only as good as its own spatial reasoning. We can test the same symmetry on the judge's verdicts. Synthetic multi-hop spatial benchmarks give exact ground truth for every intermediate relation, so step-level judging can be evaluated without human labels. That is unusual, and it makes the study feasible.

Core Claim

For spatial questions, uncertainty should be measured over answers pooled across a group of answer-preserving or answer-transforming input symmetries, not over repeated samples of one input. We call this equivariant semantic entropy. Let $G$ be a set of transformations $g$ (identity, mirror, global 90° rotation of direction words, entity renaming, sentence shuffling), each with a known map $\phi_g$ from answers on the original input to answers on the transformed input. Let $a_{g,m}$ be the $m$-th of $M$ sampled answers on $g(x)$, and let $c_1,\dots,c_K$ be canonical answer classes, such as the 8 compass directions plus "overlap". Then

$$\hat p_k = \frac{1}{|G|M}\sum_{g\in G}\sum_{m=1}^{M} \mathbb{1}\left[\phi_g^{-1}(a_{g,m}) = c_k\right], \qquad \mathrm{ESE}(x) = -\sum_{k=1}^{K}\hat p_k\log\hat p_k .$$

The method needs only answer frequencies, so it runs on closed APIs. Clustering is exact canonicalization of spatial labels rather than a learned entailment model. This matters because entailment models are known to mishandle spatial and coordinate claims.

We make three testable predictions. First, at a fixed number of model calls, ESE detects wrong answers better than ordinary semantic entropy, and the whole gain comes from errors where ordinary entropy is near zero. Second, applied per hop in a chain, ESE identifies the first wrong hop. Third, applied to a judge's step verdicts, ESE separates trustworthy from untrustworthy verdicts at constant cost, where confusion prompting needs a number of calls that grows quadratically with the rating scale. We also predict, and will report, where the method should fail. Errors that are themselves equivariant are invisible to it. The clearest case is answering in the camera's frame when the question asks for a person's frame, since mirroring transforms that wrong answer exactly as a right one would.

Relation to Prior Work

Two papers supply the method. Semantic Uncertainty introduced entropy over meaning-equivalence clusters of sampled answers. It beat token-level entropy and verbalized confidence on TriviaQA and CoQA, but it samples from a single input and inherits its blind spot: a model that hallucinates consistently gets zero entropy. The Art of Interrogation supplies the symmetry idea. It uses horizontal flips and rephrasings as label-free consistency rewards for reinforcement learning, reaching 76.8% on SUN RGB-D against 79.6% with labels. It uses consistency only pairwise, as a training signal, on single-hop relations. We use the same symmetries as a test-time uncertainty measure over a full answer distribution and extend them to multi-hop chains and to judges.

Three papers motivate the gap. StepGame shows accuracy collapsing with chain length, with its memory network falling to 31.25% at five hops, and it reports no notion of confidence. Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning shows that a symbolic solver fed by neural extraction fails when one extracted relation is wrong, because nothing flags it. Vision Language Models See What You Want but not What You See reports that scale barely helps perspective-taking. That suggests systematic rather than random errors, which is exactly the regime where dispersion-based uncertainty fails.

Two papers also constrain the design. VL-Uncertainty perturbs images with Gaussian blur and rephrasing, then clusters answers with an LLM. Blur leaves geometry untouched and only adds noise, so it cannot expose a left/right bias. We therefore treat it as a baseline, not a component. Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods warns about inconsistent metrics and prompt sensitivity. We follow it by fixing prompts across methods and reporting both AUROC and selective accuracy.

On the judging side, MLLM-as-a-Judge finds that multimodal judges hallucinate visual details when made to reason step by step, and that they do better from text descriptions than from pixels. It diagnoses the problem without filtering unreliable verdicts. Black-box Uncertainty Quantification for LLM-as-a-Judge filters verdicts by persuading the judge toward each rating. It reaches 100% accuracy on the low-uncertainty subset of Feedback Collection, at quadratic cost and on single-turn text only. AgentEval shows that step-level judging finds far more failures than outcome checks (recall 0.89 vs 0.41) but treats the judge's step verdicts as correct. Uncertainty Guided Global Memory tracks uncertainty across hops in Wikipedia QA, using token and hidden-state signals that have no notion of spatial symmetry.

Research Program

The first component is a symmetry library with exact answer maps for each benchmark. For StepGame text stories we use mirror (east and west swap), rotations of all direction words by 90°, 180° and 270°, consistent entity renaming, and shuffling of the order of distractor sentences. Each of these is an exact symmetry of the task, because StepGame relations are pure 2D grid offsets. For images on SpaCE-10 (via the SpaCE-10 codebase) and ViewSpatial-Bench, we use horizontal mirroring with relabeling of left and right. We also use question rephrasing. Counting and identity questions are invariant under both. We filter out images containing readable text or other chirality cues that a mirror would break, and report how many items remain.

The second component is hop-level scoring. For a StepGame story with a k-hop chain, every intermediate pair has a ground-truth relation computable from the grid. We query the reasoner for each intermediate relation, compute ESE per hop, and predict the first failing hop as the earliest one above a threshold tuned on a held-out split. We then test whether gating helps. When a hop is uncertain, we either re-query with more samples or hand that hop to a symbolic solver, in the spirit of the extraction-plus-solver pipeline in Disentangling Extraction and Reasoning. We measure whether end accuracy at k = 5 to 10 improves over ungated chains.

The third component is judge auditing. We collect step-by-step solutions from several reasoners on StepGame and on SpaCE-10 compositional items. A judge model grades each step as correct or incorrect. Ground truth comes from geometry for StepGame and from benchmark annotations for SpaCE-10 final answers. We then transform the step and its context together, mirroring both the image and the step's text, and ask the judge again. A step that is correct should stay correct after both are mirrored, so a verdict that changes is a sign of an unreliable judgment. ESE over the judge's pooled verdicts becomes the judge's uncertainty, with a cost of $|G|\times M$ calls regardless of rating scale.

Baselines

The naive reference point is the reasoner's or judge's single greedy answer with no uncertainty estimate, or equivalently a random abstention order. Selective-accuracy curves start from this point, and AUROC for detecting wrong answers would be 0.5.

The strongest existing uncertainty method for the reasoner is ordinary semantic entropy from Semantic Uncertainty, computed on the untransformed input. Clustering uses the same exact label canonicalization we use, so the only difference is whether samples come from transformed inputs. It is the strongest competitor because it is the method our claim modifies, and it is reported to beat likelihood and verbalized baselines. To keep the comparison fair, it gets the same total number of calls as ESE: if ESE uses five transforms with two samples each, semantic entropy gets ten samples at the same temperature (0.5, the setting reported as best there), with the same prompts and the same threshold-tuning split. No reported result found on Lacuna for semantic entropy AUROC on StepGame, SpaCE-10 or ViewSpatial-Bench. We will therefore establish it ourselves, and count ESE as beating it if it raises error-detection AUROC with confidence intervals that exclude zero difference on at least two of the three benchmarks.

For the multimodal case we also run VL-Uncertainty with blur and rephrasing at the same call budget. Its reported gain is +10.09% hallucination-detection accuracy for InternVL2-8B on MM-Vet, a benchmark that is not spatial. ESE beats it if geometric transforms outperform blur on spatial items while matching it on non-spatial ones.

The key ablation separates pooling from merely transforming. We compare three variants: per-transform entropy averaged across transforms (no pooling, so consistent wrong answers in each view still look certain), a binary flip-agreement score in the style of the pairwise check in The Art of Interrogation, and full pooled ESE. A second ablation restricts $G$ to invariances only (rename, rephrase, shuffle) versus including the transforms that change the answer (mirror, rotate). This tests our claim that the symmetries which change the answer are what expose bias.

For judges, the strongest competitor is confusion prompting from Black-box Uncertainty Quantification for LLM-as-a-Judge. Its reported result is 100% accuracy on the low-uncertainty subset of Feedback Collection. On a binary step-correctness scale its cost is small, so we run it at its native cost and also run ESE at that same call count. We also compare against verbalized judge confidence and a self-consistency vote over judge samples on untransformed input. ESE wins if, at equal calls and equal coverage, its kept verdicts are more accurate on spatial steps.

Why not just…?

Why not just use ordinary semantic entropy with more samples? More samples from one input sharpen the estimate of the model's preference on that input. They cannot reveal a preference that is biased the same way every time. Our decisive diagnostic is the subset of wrong answers where ordinary entropy is near zero. If ESE assigns those items high uncertainty and ordinary entropy at ten times the budget still does not, extra samples are not a substitute. If ordinary entropy matches ESE once its budget is matched, the proposal's central claim is false, and we will report that.

Why not just check flip agreement, which is cheaper? This is the real threat. On single-hop binary questions, one mirrored query may capture most of the signal. The ablation above settles it. We expect full pooling to matter most when answers have many classes (StepGame's 8 directions, ViewSpatial-Bench's 18 labels) and for per-hop scoring, where a graded score ranks hops better than an agree/disagree bit. If binary agreement matches ESE everywhere, we would narrow the contribution to the hop-localization and judge-auditing results and present the binary check as the recommended tool.

Why not just train the bias away, as with consistency reinforcement learning or SpatialThinker? Training on spatial objectives does raise accuracy, as SpatialThinker and the consistency-trained model in The Art of Interrogation show. But training does not tell a user which remaining answers to distrust, and it is unavailable for closed models and for judges one does not control. We will apply ESE to a consistency-trained checkpoint where one is available to us, and test whether it still detects residual errors. If training removes nearly all detectable inconsistency, ESE's value shifts to closed models and judges, and we will say so.

Why not just give the judge a text description of the scene, since MLLM-as-a-Judge found that helps? That finding in MLLM-as-a-Judge assumes an accurate description exists. On SpaCE-10 and ViewSpatial-Bench, writing one is the spatial problem itself. On StepGame the judge already receives pure text and, we expect, still errs on long chains. Judge accuracy on StepGame steps is the direct test of whether text input alone suffices.

Proposed Experiments and Decisive Checks

The reasoners are open models in the 3B to 8B range, including the Qwen2.5-VL-3B base that ViewSpatial-Bench reports at roughly 35.85%. They are joined by one proprietary model, GPT-4o, which is reported at 49.0% on SpaCE-10 and 34.98% on ViewSpatial-Bench. Text-only StepGame runs use open instruction-tuned LLMs. For uncertainty we report error-detection AUROC and selective accuracy at 50%, 70% and 90% coverage. For hops we report first-wrong-hop localization accuracy on StepGame, broken out by k from 2 to 10, along with the change in end accuracy from uncertainty-gated delegation compared with an ungated chain and with a chain that delegates every hop. For judges we report step-verdict accuracy, AUROC of judge uncertainty for predicting judge error, and accuracy of kept verdicts at matched coverage and matched calls. The kept-verdict analysis is also how we test AgentEval's assumption that judge verdicts can be trusted.

Three checks decide the proposal. First, the confident-error subset analysis: the fraction of errors with near-zero ordinary entropy that ESE flags. Second, StepGame hop localization against exact geometric ground truth, which needs no human annotation. Third, the predicted failure on ViewSpatial-Bench person-centric questions. There, frame-confusion errors are themselves equivariant under mirroring, so ESE should not detect them better than ordinary entropy. Finding this failure would confirm we understand what the signal measures. Finding a large gain instead would mean the mechanism is something other than what we claim, and we would investigate before drawing conclusions.