Vision-language models, the systems that answer questions about images by passing image features into a language model, describe the world well from the camera's point of view. They fail when asked to take someone else's. Ask "from the woman's position, is the lamp on her left?" and accuracy falls to near chance. The failure is well documented and so far has been patched from outside the model. One approach fine-tunes the language model on tens of thousands of perspective questions. Another hands the geometry to external detectors and depth estimators, which redraw the scene as a diagram. We propose a third route inside the model. We learn a small, angle-conditioned transformation of the image features that re-expresses the scene as if the camera stood where the other agent stands. The language model then only has to answer the camera-view question it already answers well.
The mechanism borrows from two literatures that have not met. Neuroscience models of parietal cortex show that brains convert eye-centered positions into other reference frames through "gain fields": neurons whose response to a stimulus is multiplied by a posture signal such as eye or head angle. In such a population, any target frame can be read out linearly (Pouget et al.; Basis function paper). Work on aligning embedding spaces shows that independently trained contrastive models are related by a single orthogonal map, recoverable in closed form (Canonicalizing Multimodal Contrastive Representation Learning). A change of viewpoint cannot be one fixed rotation, because the needed map depends on the viewpoint angle. It may, however, be a family of rotations indexed by that angle. This project tests that bet.
Core Claim
Frozen vision encoders already carry directional information that the language model underuses. 3ViewSense finds that a small probe on frozen visual features reaches 55.8% on block counting, beating the full model. VLMs also answer camera-frame spatial questions far better than other-frame ones (ViewSpatial-Bench; SpinBench).
We claim that converting an other-agent question into a camera-frame question can be done by a structured, angle-conditioned transformation of the visual tokens before they reach the decoder. We further claim that this structure, and not extra fine-tuning capacity, is what lets the model generalize to viewpoint angles it never saw during training.
Concretely, let $f_i$ be the feature of visual token $i$ after the frozen encoder and projector, and let $\theta$ be the heading of the target viewer relative to the camera. The simplest structured form is a one-parameter rotation group acting on a learned subspace:
$$f_i' = P^\top \exp(\theta A)\, P f_i + (I - P^\top P) f_i .$$
Here $P$ is a learned projection onto a $2m$-dimensional "geometry" subspace with orthonormal rows. $A$ is a block-diagonal skew-symmetric matrix with $m$ learned frequencies, so $\exp(\theta A)$ is orthogonal for every $\theta$. The complementary subspace, which carries appearance, passes through unchanged. This is the angle-indexed extension of Procrustes alignment: it preserves geometry exactly and has only one degree of freedom per plane.
The more expressive, biologically motivated variant is a gain-field adapter:
$$f_i' = f_i + \sum_{k=1}^{K} g_k(\theta)\, B_k f_i .$$
Here $g_k$ are fixed smooth basis functions of the angle (for example $\cos k\theta$ and $\sin k\theta$) and $B_k$ are learned low-rank matrices. The response to each token is therefore multiplied by posture-dependent gains, as in Hierarchical Transformation of Space in the Visual System.
The rotation-group form predicts smooth interpolation to unseen angles. The gain-field form can absorb non-rigid effects such as occlusion and objects' own facing directions, but it might overfit the trained angles. Comparing them tells us how much structure the problem really wants.
The angle $\theta$ has to come from somewhere. We therefore separate two sub-problems: inferring the reference frame (who is the viewer and which way do they face) and transforming into it. A small head, analogous to the perspective-grounding head of PATRON, predicts $\theta$ from the question text and the visual tokens. We report results with both ground-truth and predicted $\theta$. This matters because Keep It in Mind attributes 23% of its modular system's errors to ego-pose confusion. We want to know which half of the problem our method actually solves.
Relation to Prior Work
Two lines of work motivate the gap. Keep it SymPL lifts the scene to 3D with off-the-shelf detectors, depth and orientation estimators. It then redraws the scene as colored dots in the target object's frame, so the VLM answers an easy 2D question. Our logic is the same, but the re-expression happens in latent space and is trained end to end, so detector errors cannot cascade irreversibly. ViewSpatial-Bench trains MVSM by fine-tuning Qwen2.5-VL-3B's language decoder on 43,000 perspective samples with a frozen encoder. It treats the transform as linguistic reasoning and reports flip-flopping when perspectives change. We instead put a geometric prior before the decoder.
Two further papers constrain the design. The premise-based tasks in SpinBench show that models fail even when spatial facts are stated in text. This looks like a threat, since it suggests the decoder cannot manipulate spatial relations at all. We read it the other way: it argues against asking the decoder to perform the rotation, and for performing the rotation upstream so that the decoder only does the camera-frame readout it handles well. Extending Multi-modal Contrastive Representations finds that linear maps preserve topology better than non-linear ones when bridging a modality gap. This is why our default is orthogonal and the non-linear gain-field form must earn its place.
Three papers supply the method. The Procrustes result supplies the orthogonal machinery and our initialization: we first fit a single $Q$ between probe directions in camera and other-agent frames. Flow Equivariant World Models shows that building ego-motion equivariance into a latent map gives 96% spatial-probe accuracy where diffusion models get below 1%. That work used synthetic worlds without language, and we carry the equivariance idea into a VLM's token space. DSGCR splits geometry into symmetric (distance) and antisymmetric (direction) Fourier components. Our rotating planes are the antisymmetric part, while distance-like relations should be invariant under $\exp(\theta A)$. This gives a built-in check: "closer/farther" accuracy should be unaffected by the transform.
Research Program
The first stage is diagnostic. On ViewSpatial-Bench scenes, which come from ScanNet with known poses, we fit linear probes on frozen Qwen2.5-VL-3B visual tokens. The probes predict each object's direction sector in the camera frame and in the other agent's frame. We then ask whether a single orthogonal map aligns the two probe spaces, or whether the best map varies smoothly with $\theta$. If camera-frame direction is not linearly decodable at all, the project's premise fails early and cheaply, and we fall back to reporting that negative result.
The second stage trains the rotation-group adapter and the gain-field adapter, plus the $\theta$-prediction head, with the vision encoder and the language model frozen. Training data is the ViewSpatial perspective data, with $\theta$ available from scene geometry. The training loss is the standard answer likelihood, plus an optional auxiliary term asking the transformed tokens' camera-frame probe to recover the other-frame sector. The third stage evaluates transfer to benchmarks the adapters never trained on.
Baselines
The naive reference point is base Qwen2.5-VL-3B with no adaptation. On ViewSpatial-Bench it scores 35.85% overall, against 34.98% for GPT-4o and about 26% for proprietary models near chance (ViewSpatial-Bench). We also run it with a step-by-step reasoning prompt, so that prompting gets its fair chance.
The strongest learned competitor is MVSM, the decoder fine-tuned on 43,000 multi-perspective samples, which reports 82.09% overall on ViewSpatial-Bench (ViewSpatial-Bench). We retrain it ourselves on exactly the same training split as our adapters, including the held-out-angle splits described below. We give it the same number of optimizer steps and a matched hyperparameter sweep, and we also run a LoRA variant with the same trainable-parameter count as the gain-field adapter. On the standard split, beating MVSM means matching its 82.09% within noise using a small fraction of its trainable parameters. On held-out angles and on SpinBench and COMFORT#, it means a clear margin, which we pre-register as at least five absolute points.
The strongest modular competitor is SymPL. It reports 97.33% on "closer" and 91.50% on "facing" tasks in COMFORT#, where standard VLMs hover near 50% (Keep it SymPL). We run SymPL with the same base VLM. We do not expect to beat it on clean synthetic COMFORT#. The comparison that counts is on real-image 3DSRBench, where no reported result was found on Lacuna, and under controlled corruption of its orientation estimates. Staying within a few points of SymPL on synthetic data while degrading more gracefully on real data would count as success.
Two ablations isolate the key ingredient. The first uses a single $\theta$-independent orthogonal map fit by Procrustes, which tests whether the angle-dependence matters. The second uses an unconstrained MLP adapter that sees $\theta$ as an input feature, with parameters matched to the gain-field adapter. This tests whether the multiplicative or group structure matters beyond simply conditioning on angle. A LayerNorm-only tuning baseline, following Tuning LayerNorm in Attention, tests whether the gap is merely a distribution shift.
Why not just…?
Why not just fine-tune the decoder on more perspective data, as MVSM does? This is the real threat. MVSM's 82.09% shows that the in-distribution problem is largely solvable this way. Our claim is narrower: decoder fine-tuning learns the angles it sees, while a rotation group should interpolate. The held-out-angle experiment settles this. If a retrained MVSM matches our adapter on unseen angles and on SpinBench, the structural claim is false and we will report it as such.
Why not just use the modular pipeline, as SymPL does? On synthetic scenes it is nearly solved, and we concede that. The evidence against it as a general solution comes from its streaming analogue, DirectMe, where 40% of errors come from tracker loss and 23% from ego-pose errors (Keep It in Mind). The orientation-corruption experiment measures directly whether latent re-expression is more robust than a cascade that commits to hard estimates.
Why not just prompt the model to reason step by step? 3ViewSense reports reasoning chains exceeding 10,000 tokens that still end in wrong answers. SpinBench's premise-based failures show that the decoder cannot rotate spatial facts even when they are stated explicitly. We nonetheless include the prompted baseline so this is measured on our splits rather than assumed.
Why not just tune LayerNorm or LoRA? If the gap were only a distribution shift between modalities, cheap normalization tuning would close it. Perspective changes, however, reverse "left" and "right" depending on angle, which a re-centering and re-scaling cannot express for all angles at once. The parameter-matched LoRA and LayerNorm runs test this directly.
Proposed Experiments and Decisive Checks
The central experiment builds held-out-angle splits of ViewSpatial-Bench using its 18-sector discretization. We train only on examples whose relative viewer heading falls in, say, the 0°, 90° and 180° neighborhoods, and test on 45° and 135°. The prediction is specific. The rotation-group adapter should lose little accuracy on the held-out angles. The retrained MVSM and the unconstrained MLP should lose markedly more. The single-$Q$ Procrustes ablation should fail at all nonzero angles.
Transfer is measured zero-shot on three further benchmarks, with no additional training. On SpinBench's perspective-taking and rotation categories, humans score 91.2% and most of the 43 tested VLMs are near chance. On COMFORT#, the comparison is with SymPL. On 3DSRBench, the real-world test, no reported number was found on Lacuna.
Frame inference is scored separately from frame transformation, by comparing ground-truth $\theta$ against predicted $\theta$. The accuracy gap between the two quantifies how much error is due to not knowing whose perspective the question asks for.
A geometric sanity check comes from the rotation-group form's own prediction. Distance relations ("closer") should be invariant to the transform. Accuracy on them should therefore stay flat, while direction relations ("left of", "behind") improve. A result that improves both, or neither, would indicate that the adapter is doing something other than a frame change.
Finally, the orientation-robustness test injects controlled angular noise into both SymPL's orientation estimates and our predicted $\theta$, and plots accuracy against noise. This gives a direct answer to whether an internal, soft transform degrades more gracefully than an external, hard one.
References
- Hierarchical Transformation of Space in the Visual System
- Canonicalizing Multimodal Contrastive Representation Learning
- 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMs
- PATRON: Perspective-Aware Multitask Model for Referring Expression Grounding Using Embodied Multimodal Cues
- Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams
- Keep it SymPL: Symbolic Projective Layout for Allocentric Spatial Reasoning in Vision-Language Models
- Extending Multi-modal Contrastive Representations
- Flow Equivariant World Models: Structured Memory for Dynamic Environments
- DSGCR: Decomposed Spectral Geometry-Aware Cross-Modal Semantic Representation for 3D Visual Grounding
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning