Vision-language models (systems that read an image and answer questions about it in text) are weak at simple spatial judgments such as "is the mug left of or right of the plate?". They are also biased toward whichever answer option appears first. We propose to treat these as one problem. The model should compare all candidate answers side by side, the way modern document rerankers compare passages in a single prompt. It should then be trained so that its choice does not change when the candidates are shuffled, and does change in the predictable way when the image is mirrored. The shuffling constraint removes the easy positional escape route. The mirroring constraint forces the remaining signal to come from the image geometry, and it costs no human labels.
Why this matters: current spatial benchmark scores partly measure a preference for option "A". Choosing 'Right' from Wrong ran all 24 orderings of four options on the What's Up spatial-preposition set. LLaVA-1.6 (Mistral-7B) picked the first option 43% of the time against an expected 25%, and every one of eight models rejected the no-bias hypothesis at $p<10^{-4}$. The usual fix, averaging over many shuffled orderings, multiplies inference cost roughly 24-fold. It also leaves the model no better at seeing where things are.
Core Claim
Our central hypothesis is that, on spatial questions, positional bias is the fallback a model uses when it cannot read geometry from the image. This is the "blind guessing" account proposed for video question answering by Addressing Blind Guessing. If the hypothesis holds, two things follow. First, order sensitivity should cluster on the items the model gets wrong for geometric reasons. Second, training that closes the positional route should push the model toward the visual signal, provided the training data contains candidates that differ only in their spatial relation.
We therefore build a listwise spatial comparator: all candidate relations appear in one prompt, and the model outputs a distribution over them. It is trained with two consistency terms on minimal-pair candidate lists, meaning lists such as "left of / right of / on / under" for the same two objects. The first term is permutation invariance: the answer distribution must follow the candidates when their order changes. The second is flip equivariance: mirroring the image horizontally must move probability from "left of" to "right of", and from "right of" to "left of". This also applies in the frame of a person in the scene, whose left and right likewise swap under a mirror.
Let $p_\theta(\cdot \mid x, \pi)$ be the model's distribution over candidates given image-question input $x$ and candidate ordering $\pi$. Let $\sigma$ be a random reordering and let $f$ denote horizontal mirroring with its induced label swap $S_f$. The training objective adds to the usual cross-entropy $\mathcal{L}_{CE}$:
$$ \mathcal{L} = \mathcal{L}_{CE} + \lambda_1\, \mathrm{KL}\big(p_\theta(\cdot\mid x,\pi)\,\|\,p_\theta(\cdot\mid x,\sigma\pi)\big) + \lambda_2\, \mathrm{KL}\big(S_f\, p_\theta(\cdot\mid x,\pi)\,\|\,p_\theta(\cdot\mid f(x),\pi)\big) $$
Here both distributions in each KL term are indexed by candidate identity, not by slot, and $\lambda_1,\lambda_2$ are weights. The first KL term penalizes answers that depend on slot position. The second penalizes answers that ignore mirroring, which is the image change a positional or co-occurrence shortcut cannot track. Examples of such shortcuts include the "cups are on tables" prior documented in What's "up" with vision-language models?.
This differs from the closest work in three ways. Choosing 'Right' from Wrong measures the bias but proposes no training fix and never uses a comparative format. Is ChatGPT Good at Search? notes input-order sensitivity in listwise text reranking but works only on text and relies on sliding windows rather than invariance training. LamRA builds listwise multimodal reranking but targets semantic matching and does not measure order bias. Shuffle-based training such as QMOS randomizes order but has no image-side term. Without that term, a model can become invariant by being uniformly ungrounded.
Research Program
The work has three stages, and each can fail on its own.
Stage 1: diagnosis. We reproduce the 24-permutation protocol of Choosing 'Right' from Wrong on What's Up, ViewSpatial-Bench and STSnu. We run it in two formats: standard lettered multiple choice, and a listwise prompt with numbered candidates in the style of RankGPT. For each item we record order disagreement, defined as the entropy of the predicted answer across permutations. We then test whether disagreement predicts error on geometric items (left/right, allocentric) more strongly than on semantic ones. We also test whether a cheap two-ordering check (forward versus reversed) recovers most of what the full 24-ordering check captures. If order disagreement does not concentrate on geometric failures, the fallback hypothesis is wrong for spatial tasks, and the flip term has no reason to help beyond ordinary augmentation.
Stage 2: training. We train the comparator on Qwen2.5-VL-3B with a frozen vision encoder. This matches the MVSM setup in ViewSpatial-Bench, so the strongest baseline shares our backbone. Training data comes from automatically generated spatial questions produced with the ViewSpatial recipe (43,000 samples), plus STVQA-7K from SpatialThinker. Every question is expanded into minimal-pair candidate lists. Mirroring is applied only to relations whose labels transform cleanly under a flip. Mirrored images containing text or strongly handed objects are filtered out.
Stage 3: the reasoning question. We test whether step-by-step reasoning helps before ranking in this setting. The text literature disagrees. Rank-K reports large gains from reasoning traces at 32B. Rethinking Reasoning in Document Ranking reports that reasoning hurt listwise ranking at 8B (BRIGHT nDCG@10 of 25.9 with reasoning versus 27.1 without) and worsened calibration. At our 3B scale we run the comparator with and without a reasoning prefix. We report both accuracy and permutation consistency, because Serial Position Effects of Large Language Models found that step-by-step reasoning reduced position effects, which means reasoning could help consistency even if it does not help accuracy.
Baselines
Untuned model. The naive reference point is untuned Qwen2.5-VL-3B, prompted zero-shot in lettered multiple-choice format with a single fixed ordering. ViewSpatial-Bench implies it scores about 35.85% on ViewSpatial-Bench, since MVSM's 82.09% is reported as a 46.24-point gain over the base. For scale, GPT-4o is reported there at 34.98%. This baseline shows how much of any gain is simply fine-tuning.
MVSM, the strongest existing method. MVSM is the multi-perspective fine-tune from ViewSpatial-Bench, reported at 82.09% accuracy on ViewSpatial-Bench. It is the right competitor because it uses the same backbone and the same kind of automatically generated 3D data, and it is the largest reported gain on allocentric questions. We retrain it ourselves on exactly our data mixture, with the same number of optimizer steps and the same hyperparameter search budget. We also check that our reproduction lands near 82.09 before comparing.
Accuracy alone will not count as beating MVSM, because it may already be near the benchmark's ceiling. Our success criteria are three:
- Matching MVSM's single-ordering accuracy within noise while cutting first-slot selection to within a few points of 25%.
- Substantially raising the fraction of items answered identically across all 24 orderings.
- Outperforming MVSM on the held-out What's Up set and on STSnu Agent-to-Agent items, where the flip term's geometric pressure should matter most.
For What's Up, the best prior accuracy in the file is 56%, from BLIP fine-tuned on VQAv2 (What's "up"). No reported result was found on Lacuna for any debiased listwise model on What's Up. For STSnu, the file reports InternVL2.5-8B at 46.07% and text-only GPT-4o given ground-truth coordinates at 57.08% (STSBench). We treat the latter as a soft ceiling for what reading geometry correctly could achieve.
Test-time debiasing of MVSM. We apply two corrections to the trained MVSM model. The first is permutation ensembling over all 24 orderings. The second is Weighted_BOLD calibration from Addressing Blind Guessing, which reported up to 7.94% on STAR. We run it with its 5-fold cross-validated fitting on our validation split. These baselines test whether a post-hoc correction on top of the strongest model is enough. We report their compute cost next to their accuracy.
Ablations. These isolate the key ingredient, each trained with the same data and budget:
- Listwise training with shuffling only, the QMOS-style arm ($\lambda_2=0$).
- Listwise training with mirroring used as plain data augmentation and no consistency term.
- A pointwise yes/no verifier, trained on the same minimal pairs with the same flip augmentation, that scores each candidate separately.
Why not just…?
Why not just shuffle at test time and vote? Ensembling over 24 orderings costs 24 forward passes per question. Choosing 'Right' from Wrong also shows that averaging over orderings exposes the bias but does not add spatial competence. We include full ensembling as a baseline. If our single-pass model matches ensembled MVSM on accuracy and consistency at one twenty-fourth of the cost, the proposal has practical value even if the accuracy gain is small.
Why not just score each option separately, which has no order at all? This is the most serious threat. A pointwise verifier trained with the same mirrored data could match us. The evidence against it is indirect. It comes from text: Differentiating Choices via Commonality shows that scoring options jointly lets a model discount what they share. Rethinking Reasoning in Document Ranking shows pointwise scorers becoming miscalibrated, with ECE rising from 0.105 to 0.151 under reasoning. Neither result is spatial. The pointwise ablation settles it directly. If the pointwise verifier matches the comparator on What's Up and ViewSpatial, we will conclude that the flip consistency term, not the listwise format, carries the gain, and report it that way.
Why not just calibrate the outputs with BOLD or OTTER? Both adjust output scores without changing what the model sees. BOLD assumes a uniform answer distribution (Addressing Blind Guessing). OTTER needs a batch and an estimate of the target label distribution (OTTER). The STSnu driving scenarios are unlikely to have uniform answers. The decisive test is the mirrored-image evaluation. A calibrated model whose accuracy does not track mirroring has only been rebalanced; it has not been grounded.
Why not just use a bigger model? Vision Language Models See What You Want but not What You See reports essentially no improvement in perspective-taking with scale. GPT-4o sits near chance on ViewSpatial-Bench. Choosing 'Right' from Wrong finds the first-option preference persisting in Llama-3.2-Vision 11B. We include GPT-4o under the 24-ordering protocol as a reference point.
Proposed Experiments
Evaluation uses held-out What's Up (not used in training), ViewSpatial-Bench, and STSnu broken down by interaction type. For each system we report five measures:
- Single-ordering accuracy.
- First-slot selection rate, with the chi-square test from Choosing 'Right' from Wrong.
- The fraction of items answered identically across all 24 orderings.
- Mirror-consistency, meaning the fraction of left/right items whose answer swaps correctly when the image is mirrored.
- Accuracy when the correct option is replaced by "none of the above".
The last measure adapts the check from LLMs May Perform MCQA by Selecting the Least Incorrect Option. That paper showed that stable answers can still come from picking the least wrong option, so it guards against a model that is consistent without being grounded.
Three results would decide the proposal. First, Stage 1 must show that order disagreement predicts geometric errors; otherwise the premise fails. Second, the full objective must beat shuffle-only training on mirror-consistency and none-of-the-above accuracy. That would show the image term, not invariance alone, supplies the grounding. Third, the comparator must beat or match the pointwise verifier trained on identical data; if it does not, we drop the listwise claim. A secondary check is whether the untrained two-ordering disagreement signal flags wrong answers well enough to decide when to fall back to 24-ordering ensembling. Such a cheap router would be useful even if the training gains turn out modest.
References
- Choosing 'Right' from Wrong: A Closer Look at Selection Bias in Spatial Multiple-Choice Questions in Large Multimodal Models
- Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
- LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant
- QMOS: Enhancing LLMs for telecommunications with question-masked loss and option shuffling
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards
- Rank-K: Test-Time Reasoning for Listwise Reranking
- Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short
- Serial Position Effects of Large Language Models
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving
- Differentiating Choices via Commonality for Multiple-Choice Question Answering
- OTTER: Effortless Label Distribution Adaptation of Zero-shot Models
- Vision Language Models See What You Want but not What You See
- LLMs May Perform MCQA by Selecting the Least Incorrect Option