Ask a vision-language model (a system that answers questions about images) whether a cup is on the person's left, and it will usually tell you where the cup sits on the image's left. This failure is called perspective locking. Across twelve models, answering from another agent's viewpoint costs roughly 35 points of accuracy relative to matched control questions, often leaving models near chance (Seeing Through Their Eyes). The usual fix is to collect tens of thousands of 3D-annotated multi-view examples and fine-tune on them (ViewSpatial-Bench). This proposal takes a different view, borrowed from search-engine click modeling. In web search, a user clicks a result only if they first look at it and then judge it relevant. The position of a result drives the looking, so clicks mix "was it seen" with "was it good." We propose that a model's spatial answer has the same structure. It first silently chooses a frame of reference, camera or agent, and then computes the geometry within that frame. The camera frame plays the role of the top-ranked slot: it gets chosen too often, whatever the question asks.
Treating the problem this way makes it measurable and, under stated conditions, correctable without training. Click modeling separates looking from relevance by deliberately swapping result positions (Accurately interpreting clickthrough data). We separate frame choice from geometric skill by deliberately turning the agent around. That lets us estimate, per model and per condition, how often it silently answers in the camera frame. We then subtract that camera-frame component out of the model's answer distribution, using a second forward pass that asks the camera-frame question explicitly.
Core Claim
Consider a directional or visibility question $q$ about agent $g$ in image $x$. We model the answer distribution as a two-component mixture:
$$P(a \mid x, q) = \pi(x)\,P_{\text{cam}}(a \mid x) + (1-\pi(x))\,P_{\text{agt}}(a \mid x)$$
Here $a$ is a candidate answer, $\pi(x)$ is the probability that the model silently adopts the camera frame, $P_{\text{cam}}$ is its answer distribution when reasoning in the camera frame, and $P_{\text{agt}}$ is its answer distribution when reasoning in the agent's frame.
The mixture weight $\pi$ can be identified from data. When the agent faces away from the camera, the camera-frame and agent-frame answers to a left/right question coincide. When the agent faces the camera, they are opposite. Accuracy on the aligned items therefore measures geometric skill alone, while accuracy on the opposed items mixes skill with frame choice. That difference pins down $\pi$, just as rank-swap interventions pin down examination probability in click models.
Once $\pi$ is estimated, the agent-frame answer can be recovered. We read $P_{\text{cam}}$ from an explicit camera-frame query on the same image and compute
$$\hat P_{\text{agt}}(a \mid x) \propto \max\!\left(0,\; P(a \mid x, q) - \hat\pi(x)\,P_{\text{cam}}(a \mid x)\right)$$
The estimate $\hat\pi(x)$ may depend on observable conditions: the agent's facing angle, its image location, and the number of agents present. This mirrors conditional and personalized propensity models from unbiased learning to rank (Addressing Personalized Bias for Unbiased Learning to Rank).
The ranking literature also supplies an honest warning. The correction divides by $1-\hat\pi$, so when a model is almost fully locked ($\pi \to 1$) it becomes high-variance and eventually uninformative. This is exactly how inverse propensity weights explode at low-examination ranks. Whether the correction works therefore depends on an empirical question: does the agent-frame computation exist inside the model at all? We test that directly.
Relation to Prior Work
Three papers motivate this framing by showing that "bias" in vision-language models is often a prior layered over intact perception. Logit-Attention Divergence shows that models attend to the correct image in a multi-image gallery but output a favored slot, first or last. Correcting with attention-guided calibration raises 8-image retrieval accuracy for LLaVA-OneVision from 31.56% to 94.92%. That work's "positions" are prompt slots, and its method needs white-box access to attention weights. Our "position" is a reference frame, and our correction needs only answer probabilities, which can be estimated by sampling from API-only models. From Bias to Balance traces performance swings across a 3×3 image grid to the one-directional (causal) attention of the language model. Its global-context injection method cuts accuracy variance for Qwen2.5-VL-72B from 0.95 to 0.12. That analysis stays in the 2D image plane; we borrow it as one covariate of $\pi$, the agent's position in the order the image is read, and as a baseline. Seeing Through Their Eyes averages over answer orderings to remove option-order bias. It treats bias only as an evaluation nuisance and never models the frame choice itself.
Two lines of work constrain the design. ViewSpatial-Bench attributes locking to web images whose captions describe "left" as image-left, and reports that models flip frames partway through generation. This is why we score short constrained answers rather than free text, and why a single static correction applied at the start of generation is unlikely to suffice. Unbiased Learning to Rank Meets Reality shows that position-debiasing methods that work in simulation fail to beat an untuned BM25 baseline on expert-labeled real search data, because real behavior departs from the assumed model. That result forces us to test whether the two-component mixture actually fits real scenes, not only rendered ones.
Two approaches supply competing methods rather than components. The first is supervised multi-view fine-tuning (ViewSpatial-Bench). The second is a pipeline that bypasses the vision-language model's geometry entirely by building a symbolic top-down map from a detector, a depth estimator and an orientation estimator (Keep it SymPL). PATRON shows that explicitly predicting which frame is in use helps downstream grounding. This supports our premise that frame choice is a separable latent variable, although PATRON learns it with supervision.
Research Program
The first component is a controlled intervention set. We render scenes with a humanoid agent and objects and vary only the agent's facing angle in 45° steps, including exact 180° turnarounds. Each scene is paired with its horizontal mirror image. A turnaround flips the agent-frame answer but not the camera-frame answer; mirroring flips both. Mirroring is therefore a consistency check, and turnarounds are the treatment. Every item carries three questions: the agent-frame question, the matching explicit camera-frame question, and a pure perception control (does the object exist?). The set covers left/right questions (Level-2 perspective taking, meaning how the scene appears from the agent's side) and visibility-counting questions in the style of Isle-Dots (Level-1 perspective taking, meaning what the agent can see). For counting, "camera count" and "agent count" are the two mixture components.
The second component is estimation. For each model we fit $\pi$, the camera-frame and agent-frame geometric accuracies, and a small logistic model of $\pi$ given facing angle, raster position, and agent count. Fitting uses expectation-maximization over the aligned and opposed items, exactly as in position-based click models. The fitted mixture is itself a diagnostic that nobody has reported: it tells us whether a model's 35-point perspective gap comes from choosing the wrong frame or from bad geometry.
The third component tests whether the agent-frame computation exists inside the model. On open models (Qwen2.5-VL-3B/7B, InternVL2.5-8B, LLaVA-OneVision) we train linear probes on intermediate hidden states to predict the agent-frame answer on opposed items. If probe accuracy is high while output accuracy sits near $1-\pi$, the model knows the answer but reports the camera frame, and unmixing should work. If probes are also at chance, the model genuinely lacks the computation, and no inference-time method can help.
The fourth component applies the correction. We use $\hat\pi$ estimated on a calibration split of at most a few hundred items, reusing the frugal calibration spirit of Logit-Attention Divergence. We clip $\hat\pi$ to control variance and transfer it to held-out benchmarks without refitting the geometry terms.
Baselines
The naive reference is the unmodified model, answering a constrained-format question with answer options averaged over orderings as in Seeing Through Their Eyes. Averaging over orderings removes option-order effects, so any remaining gap is due to frame choice. All inference-time methods run on the same checkpoints and question sets. Each is allowed the same number of forward passes as our method (two per item), and hyperparameters are tuned on the same calibration split.
The prompting baseline is step-by-step chain-of-thought that asks the model to first state the agent's facing direction and then reason. It is the cheapest serious competitor. Beating it means higher accuracy on opposed items with no loss on aligned items.
The static-calibration baseline subtracts a fixed per-model answer prior estimated on content-free inputs. It isolates whether per-image conditioning matters. Logit-Attention Divergence argues that positional bias is input-dependent, so static subtraction should lose. If it does not, our conditional machinery is unnecessary.
The contrastive-decoding ablation scores answers by $\log P(a \mid x,q) - \lambda \log P_{\text{cam}}(a \mid x)$, with $\lambda$ tuned on the calibration split. This is the obvious "divide out the camera" heuristic. It uses the same second forward pass but no mixture model. It isolates whether the identified mixture weight, rather than just the extra camera query, drives any gain. Two further ablations are a constant-$\pi$ version, which tests whether conditioning $\pi$ on facing angle and position helps, and a mismatched-camera-pass control, which uses $P_{\text{cam}}$ from a different image and so should destroy the gains.
The strongest existing trained method is the multi-view fine-tuned model from ViewSpatial-Bench. It reports 82.09% overall on ViewSpatial-Bench, a 46.24-point absolute gain over its Qwen2.5-VL-3B base (implying about 35.9% for the base), while GPT-4o scores 34.98% and InternVL2.5-8B scores 43.24%. We do not expect a training-free method to match 82.09% on that benchmark's own distribution. On ViewSpatial-Bench, success means recovering at least a third of the fine-tuning gain on its person-perspective questions with zero training. On Isle and COMFORT#, which the fine-tuned model was not trained on, success means matching or exceeding it. We also apply our correction on top of the fine-tuned model to see whether any residual camera-frame mixture remains.
The strongest pipeline method is SymPL (Keep it SymPL), which reports 97.33% on "closer" and 91.50% on "facing" allocentric tasks on COMFORT#, where plain models sit near 50%. Allocentric here means judged relative to some object in the scene rather than the camera. We run it with its published estimators on the same items. We do not expect to beat it on clean synthetic scenes. The comparison of interest is on 3DSRBench and cluttered real scans, where upstream estimation errors should hurt SymPL. On 3DSRBench no reported result was found in the research file for any method.
For the raster-order covariate we compare against global-context injection from From Bias to Balance. If that method alone erases the dependence of $\pi$ on agent position, the causal-attention account explains that part of the problem.
Why not just…?
Why not just prompt the model to reason step by step from the agent's viewpoint? Ziarko et al. found chain-of-thought helped some models (Gemini 1.5 Flash) and hurt others (Claude 3 Opus) on perspective taking (Seeing Through Their Eyes). Prompted frontier models remain near chance on ViewSpatial-Bench (ViewSpatial-Bench). Our mixture fit adds a sharper test than overall accuracy: whether chain-of-thought lowers $\pi$ or merely changes geometric accuracy. If it lowers $\pi$ to near zero on open models, prompting is the answer and our correction is redundant, and we will report that.
Why not just fine-tune on multi-view data? This is the real threat. The 82.09% result shows fine-tuning works in-distribution. Our case rests on three points: our method needs no 3D annotation, it works on API-only models, and it is uncertain whether fine-tuning removes the camera prior or only learns benchmark-specific cues. The orientation-swap set answers the last point. If the fine-tuned model shows $\pi \approx 0$ on rendered scenes outside its training distribution, fine-tuning has genuinely fixed frame choice. In that case our contribution narrows to diagnosis and to black-box models.
Why not just use a symbolic pipeline like SymPL? It is near-perfect on clean synthetic scenes and should win there. However, it chains four models, cannot recover from upstream detection, depth or orientation errors, and never uses the vision-language model's own perception. The decisive comparison is on real scenes, 3DSRBench and ScanNet-derived ViewSpatial items. We also stratify by orientation-estimator confidence, which tells us where each approach fails.
Why not just subtract a fixed prior or average over permutations? Averaging over option orderings cannot remove a frame prior, because the prior attaches to the frame, not to option slots. The static-subtraction baseline tests the fixed-prior version directly. The mid-generation frame flipping reported in ViewSpatial-Bench is a further reason to expect static subtraction to fail on free-form outputs.
Proposed Experiments and Decisive Checks
The first decisive check is model fit. On the rendered intervention set, the two-component mixture must predict opposed-item accuracy from aligned-item accuracy and the fitted $\pi$ within confidence intervals, and it must be consistent under mirroring. If it fails the way position-based click models failed on Baidu-ULTR (Unbiased Learning to Rank Meets Reality), the framing is wrong and we report that as a negative result.
The second check is the probe-versus-output gap. It determines in advance for which models unmixing can help. The prediction is that correction gains track that gap across models.
The third check is transfer. With $\hat\pi$ fitted only on rendered scenes, we measure corrected accuracy on the Isle-Bricks and Isle-Dots perspective questions (where plain models sit at 50–60%), on the person-perspective split of ViewSpatial-Bench, on COMFORT#, and on 3DSRBench. For all of them we report accuracy on opposed and aligned items separately, because a method that flips every answer would inflate opposed-item accuracy while destroying aligned-item accuracy.
The fourth check tests the variance warning. We plot correction gain against fitted $\pi$. The prediction is gains that peak at intermediate locking and vanish as $\pi \to 1$, with clipping trading bias for variance as it does for inverse propensity weights.
Together these checks determine whether perspective locking is, like position bias in search, a correctable prior sitting on top of intact geometric competence, or a missing capability that only training or external geometry can supply.
References
- Seeing Through Their Eyes: Evaluating Visual Perspective Taking in Vision Language Models
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Accurately interpreting clickthrough data as implicit feedback
- Addressing Personalized Bias for Unbiased Learning to Rank
- Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration
- From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
- Unbiased Learning to Rank Meets Reality: Lessons from Baidu's Large-Scale Search Dataset
- Keep it SymPL: Symbolic Projective Layout for Allocentric Spatial Reasoning in Vision-Language Models
- PATRON: Perspective-Aware Multitask Model for Referring Expression Grounding Using Embodied Multimodal Cues