Decoupling Gaze, Head, and Body to Diagnose and Fix Perspective Taking in Vision-Language Models

Vision-language models, which answer questions about images, fail at a task that children solve easily: saying what is to the left of a person sitting across the table, or whether that person can see an object behind a box. Several benchmarks show they answer from the camera's point of view even when asked to take someone else's. This proposal argues that part of the failure comes from how the training and test data are built. In ordinary images the camera's frame, the depicted person's gaze, their head direction, and their body direction mostly line up. A model can therefore score well by always answering from the camera, or from whatever 2D cue happens to correlate with the right answer, without ever learning which frame a question refers to.

Neuroscience faced the same confound and solved it with a standard design. It rotates the eyes, head, and body independently of one another and then asks which coordinate system a neural response follows. Applied to vision-language models, the same design gives two things. The first is a diagnostic that measures, for any model, which frame it actually answers in. The second is training data in which no shortcut survives. On that data we test a narrow architectural hypothesis taken from parietal-cortex models: a multiplicative "gain" conditioning on the queried frame should beat plain fine-tuning on the same data, but only for questions that chain two frame changes.

Core Claim

The claim has three parts. First, perspective-taking errors in vision-language models can be broken down quantitatively into "which frame did the model answer in." This becomes measurable once the candidate frames are made to disagree factorially: camera, observer torso, observer head, observer gaze, and gravity. Second, fine-tuning on factorially decoupled scenes removes the 2D shortcuts that sank earlier fine-tuning attempts. Third, the remaining hard cases are compositions such as "left of where she is looking, given her body faces elsewhere." These are where a multiplicative frame conditioning helps and additive conditioning does not.

The new element is a pair: articulated observers whose eyes, head, and torso are rotated independently, combined with a frame-attribution analysis. No benchmark in the file separates these nested frames for depicted agents. Each treats an observer as having one orientation.

Prior work and where this sits

The motivating evidence is the "perspective gap." SpinBench evaluates 43 models on rotation and perspective tasks and finds they default to the camera viewpoint, with humans at 91.2%. Seeing Through Their Eyes reports a 35% average drop from object detection to Level-1 perspective taking, meaning judging what another viewer can see. The same paper finds only a 0.10 Spearman correlation between the two abilities. ViewSpatial-Bench traces this to camera-centric captions in web data.

These benchmarks diagnose the failure but do not say which frame a model uses instead. Our diagnostic fills that gap. Some benchmarks also omit things we include: SpinBench leaves out vertical relations, and both Isle and The 3D-PC test visibility only.

One result constrains the design. The 3D-PC shows that fine-tuned vision models reach human level on line-of-sight visibility through shortcuts like relative size and 2D placement. They then collapse to 66% on its strategy split, where humans stay near 87%. We read this as a data-confound failure, not proof that fine-tuning cannot work. That reading is the hypothesis our factorial design tests.

Do Vision-Language Models Represent Space and How? shows models cannot tell egocentric, allocentric, and object-centered frames apart when language is ambiguous. We avoid that confound by always naming the frame explicitly. Any remaining errors are then about computation, not interpretation.

Two works supply competing methods. Keep it SymPL avoids internal geometry altogether. It detects objects, estimates depth and one orientation per object, and redraws the scene as a 2D symbolic map aligned with the reference object, reaching 97.33% on "closer" and 91.50% on "facing" tasks of COMFORT#. REP lifts YouRefIt images to 3D, places the origin at the human sender, and defines axes from body pose and pointing. Both assume a single orientation per agent. Neither can represent a person whose gaze diverges from their torso. That is exactly the case the biological decoupling literature isolates, for example torso-anchored straight-ahead gain in early visual cortex (An egocentric straight-ahead bias in primate's vision) and coexisting eye- and body-centered codes in one area (Eye-Centered Representation of Optic Flow Tuning in the Ventral Intraparietal Area).

The architectural hypothesis comes from gain-field models. In these models, responses stay anchored in one frame but are multiplicatively scaled by eye or head position. This lets a downstream linear readout recover location in any frame (Hierarchical Transformation of Space in the Visual System; Pouget et al.). It contrasts with the strictly additive composition of egocentric and allocentric successor representations in Composing egocentric and allocentric maps for flexible navigation, which was shown only in gridworlds. Our multiplicative-versus-additive ablation is a direct test of that distinction in a visual-linguistic setting.

Research Program

The scene generator. We build a procedural scene generator with articulated humanoid observers. Each scene samples five quantities independently: camera yaw and pitch relative to the room, observer torso heading, head yaw relative to the torso, gaze relative to the head, and an optional gravity tilt of the rendered room. The tilt brings in the vertical frame that SpinBench omits and that Spatial memory for vertical locations. separates in humans. Objects and occluders are placed around the observers.

Question types. Each question names its frame explicitly, as in "relative to her body," "relative to where she is looking," "from the camera," or "with respect to the floor." Questions cover three kinds of judgment. Level-1 questions ask about visibility, which depends on gaze and occluders. Level-2 questions ask about left, right, front, and behind, computed in the named frame. Composition questions require two frame changes, such as the torso frame of observer A expressed relative to observer B's gaze. Multi-agent scenes are included because they are where SpinBench and Isle models conflate viewpoints.

Removing shortcuts. Answers are balanced so that, for every question, the answers implied by the other candidate frames are statistically independent of the correct one. 2D cues are also controlled: target image position, relative size, and observer-to-target pixel distance must not predict the answer.

The frame-usage profile. For each question we compute the answer $a_k$ implied by each candidate frame $k$. We then fit how a model's chosen answer $a$ depends on those frame predictions:

$$p(a) \propto \exp\Big(\sum_k w_k \,\mathbb{1}[a = a_k] + v^\top c_a\Big)$$

Here $w_k$ measures how strongly the model follows frame $k$, and $c_a$ is a vector of 2D shortcut cues for answer $a$ with weights $v$. Because the design decorrelates the $a_k$, the weights are identifiable. The same logic tells retinotopic from spatiotopic maps in Maps of Visual Space in Human Occipital Cortex Are Retinotopic, Not Spatiotopic. A model that has truly learned perspective taking puts its weight on the queried frame and near zero on $v$.

We also train linear probes on hidden states to decode target direction in each frame. This separates two failure modes: "the frame is represented but not read out," which would vindicate the "logician, not geometer" reading of SymPL, versus "never represented."

Training conditions. We fine-tune one open model, Qwen2.5-VL as used in the SymPL comparison, with low-rank adapters. Low-rank adapters are small trainable weight updates added to a frozen model. There are three conditions with identical sample counts and compute:

  • confounded data, where observer gaze, head, and torso are aligned and correlated with the camera as in natural images;
  • decoupled data;
  • decoupled data plus a frame-conditioning module.

In the module, an embedding $e$ of the queried frame and the observer's estimated part orientations scales visual tokens $h$ as $h' = h \odot (1 + g(e))$. The additive control instead computes $h' = h + b(e)$. Here $g$ and $b$ are small networks with matched parameter counts.

Baselines

Naive reference points. These are the untouched model with zero-shot prompting and the same model with chain-of-thought prompting. The prompt walks through "stand where she stands, face where her body faces." This tells us how much any training adds. The file reports that out-of-the-box models sit near 50–53% on 3D-PC visibility and near chance on the hardest SpinBench perspective tasks. Isle reports 50–60% on its perspective tasks.

Strongest existing method. This is the SymPL pipeline from Keep it SymPL. It reports 97.33% on COMFORT# "closer" and 91.50% on "facing." It is the strongest competitor because it solves the geometry outside the network, and on single-orientation objects it is close to ceiling.

To be fair we run two versions. The first is the published version, with one orientation per reference entity from OrientAnything. The second, SymPL-plus, is our extension: a skeleton and gaze estimator supplies separate torso, head, and gaze vectors, and the projection uses whichever frame the question names. Both use the same downstream model. SymPL-plus is the version the end-to-end claim must beat.

On YouRefIt we also compare with REP, at 45.7% [email protected], and the original YouRefIt model, at 54.7% accuracy at IoU 0.25 against a human level of 94.2%.

Isolating ablations. Confounded-versus-decoupled fine-tuning isolates the data claim. Multiplicative-versus-additive conditioning isolates the gain-field claim. All trained variants get the same number of examples, steps, and hyperparameter sweep.

What counts as success. Beating the naive references means at least matching SymPL-plus on held-out benchmarks without an external geometry pipeline at test time. It also means clearly exceeding the 66% that fine-tuned models reach on 3D-PC VPT-Strategy, after training only on our generator so that the result is a transfer test. The gain module counts as earning its place only if it beats additive conditioning on composition questions while tying on single-frame questions.

Why not just…?

Why not just scale the model or prompt it to reason step by step? The file's evidence is consistent against this. Frontier models still lose about 35% on Isle perspective tasks. Chain-of-thought helps some models and hurts others (Seeing Through Their Eyes). Models fail even when SpinBench supplies the spatial premises in text. We include prompting as a baseline anyway. The frame-usage profile will show whether prompting moves weight off the camera frame or merely adds noise.

Why not just use the SymPL-style toolchain? This is the real threat, and we treat it as one. On objects with a single facing direction it is near ceiling, and our method may not beat it there. Its weak points are two. It assumes one orientation per entity. It also passes upstream estimator errors through unchanged.

SymPL-plus removes the first weakness, so the decisive comparison is SymPL-plus against our decoupled model on composition questions and on naturally photographed people with divergent gaze and body. If SymPL-plus matches us there, we will report that the contribution is the diagnostic and the data, not end-to-end learning. That outcome is still useful, because it would show the pipeline needs part-level orientation and that current benchmarks hide this.

Why not just fine-tune on more synthetic perspective data, without any decoupling? The 3D-PC is the cautionary result: fine-tuned models found 2D shortcuts and fell to 66% under perturbation. Our confounded-data condition is exactly this alternative at matched scale, with only the factorial structure removed. If it does as well as decoupled data on held-out tests, our data claim is wrong.

Why not just add the frame as an extra input token instead of a multiplicative module? That is the additive control. If additive and multiplicative tie on composition questions, we drop the architectural claim and keep the data claim.

Proposed Experiments

Experiment 1: diagnostic only. We profile the zero-shot model, the chain-of-thought model, SymPL, and SymPL-plus on the decoupled test set. We report $w_k$ and $v$ per question type. The prediction is heavy camera weight for vanilla models. For published SymPL, the prediction is torso-or-head confusion whenever the two diverge, since its single orientation estimate must pick one.

Experiment 2: transfer to existing benchmarks. We train the three conditions on generator data only. Each is then evaluated without further tuning on SpinBench perspective-taking subsets, 3D-PC VPT-basic and VPT-Strategy, Isle-Bricks and Isle-Dots, COMFORT#, and ViewSpatial-Bench. No reported model number was found on Lacuna for ViewSpatial-Bench or 3DSRBench, so for those we report our own runs of every baseline. YouRefIt is run as a separate fine-tuned comparison with REP and the original model under their reported metrics.

Experiment 3: probing. Probes test whether the queried-frame target direction becomes linearly decodable after decoupled training. They also test whether decodability already exists in the untuned model, which would point to a readout failure.

Decisive checks. The project's claims hold if three things happen. Frame weight moves to the queried frame and shortcut weight $v$ approaches zero after decoupled but not after confounded training. Decoupled training clearly exceeds 66% on 3D-PC VPT-Strategy without any 3D-PC training data. Multiplicative conditioning beats additive only on two-change composition questions.

They fail cleanly if the confounded condition matches decoupled transfer, or if SymPL-plus dominates everywhere. In either case we say so and publish the generator and diagnostic as the result.