Matched-Pixel Tests for Perspective Taking in Vision-Language Models

Vision-language models (systems that answer questions about images in text) fail badly when asked what a scene looks like from someone else's point of view. Recent fine-tuning recipes appear to close much of that gap: one 3-billion-parameter model jumps to 82% on a multi-perspective benchmark. Psychophysics offers a warning about results like this. Repeatedly, an apparent "high-level" ability has turned out to ride on low-level image statistics that differed between conditions. We propose to bring that discipline to perspective taking. The test is built from minimal pairs of rendered images whose pixels are identical everywhere except the observing agent. The agent's own patch is rotated, and its brightness, contrast, and spatial-frequency content are equalized, so that only its facing direction changes. A model passes an item only if it flips its answer exactly when the agent's viewpoint flips, and it should fall to chance when the agent is removed.

The core claim is testable and falsifiable. Part of the accuracy that current methods report on perspective-taking benchmarks can be reached without any change of reference frame, using the camera's own frame, scene-layout priors, or image statistics. Fine-tuning on unmatched multi-perspective data may teach these shortcuts rather than the transformation. Matched-pixel minimal pairs give a direct measure of real perspective taking. Training on such pairs is the natural fix to test.

Motivation and Prior Work

Two results from the stimulus-normalization literature motivate the design. When luminous flux (total black area) was decoupled from topology in 5 ms discrimination tasks, the supposed early sensitivity to topology largely disappeared Topological perception: Holes in an experiment. A pair differing only in topology with matched flux was discriminated at 33.6%, below the 44.4% for a topologically identical pair that differed in flux. VanRullen's review shows that linear classifiers on Fourier power spectra alone categorize natural images at 80–90% Four Common Conceptual Fallacies in Mapping the Time Course of Recognition. The lesson is to measure the shortcut ceiling before crediting a high-level process.

We also reuse the standard toolkit as method. This includes luminance and spectral equalization of the SHINE type, as used for decoding analyses in Decoding Trans-Saccadic Memory. It also includes the strategy of statistically partialling out features from a deep network How face perception unfolds over time.

On the spatial side, the failure itself is well documented. Across 37 models, none exceeded the ~29% chance level on the Three-Mountain-style PerspectBench, and scaling did not help Vision Language Models See What You Want but not What You See. Level-1 visibility judgments drop about 35% relative to camera-view controls Seeing Through Their Eyes. GPT-4o falls from 63.25% to 27.5% when multi-hop questions switch from the camera's perspective to a human's An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models.

Three existing approaches are closest, and we do something different from each.

COMFORT renders Blender scenes with continuous 10° rotations Do Vision-Language Models Represent Space and How?. It finds near-chance consistency between "left" and "not right," and that changing an object's color or size shifts the model's spatial confidence. That second finding is direct evidence that appearance leaks into spatial judgments. However, COMFORT does not hold the image fixed while flipping the correct answer, does not equalize appearance, and does not audit fine-tuned models.

MVSM from ViewSpatial-Bench fine-tunes the language component on 43,000 multi-perspective samples. Its gains are what we audit.

SymPL Keep it SymPL and TEP Walk in Others' Shoes with a Single Glance take a third route. They avoid the problem by replacing the image with a symbolic top-down map built from a chain of external detectors and depth estimators. This works, but it says nothing about whether the model itself can perspective-take.

Research Program

The first stage is a shortcut audit of existing benchmarks: ViewSpatial-Bench, PerspectBench, Isle, and COMFORT#. For each benchmark we compute three reference policies.

The first policy is the camera-frame policy, which answers as if the question were asked from the camera's viewpoint. The second is a low-level probe. This is a linear classifier on global and per-quadrant luminance, RMS contrast, and Fourier power. It is trained on half the items to predict the answer label and tested on the other half. The third is an agent-blind probe. It is a linear classifier on frozen vision-encoder features of images in which the agent has been inpainted away.

Any accuracy these policies reach above chance is shortcut ceiling. We then re-score MVSM and the other models on the subset of items where all three policies are wrong.

The second stage builds the matched-pixel set, called MP-Persp. We extend COMFORT-CAR-style Blender scenes and Three-Mountain layouts. For each scene we render the agent at 36 facing angles, composite each agent render into an otherwise identical background, and equalize the agent patch's luminance histogram and amplitude spectrum across angles.

Two kinds of minimal pair are drawn from these renders. In flip pairs, the correct agent-frame answer changes while the camera-frame answer stays fixed. In hold pairs, the answer stays the same despite a change in the agent's angle. Answers are counterbalanced so that the camera-frame and agent-frame answers are uncorrelated. Each scene also has an agent-absent version. We will release roughly a few thousand pairs across left/right, front/back, and Level-1 visibility questions.

The third stage tests the remedy. Using the same base model, budget, and sample count as MVSM, we fine-tune on matched minimal pairs. The pairs are presented together, and a consistency term penalizes giving identical answers to the members of a flip pair. We then test transfer to the unmatched public benchmarks.

Baselines

The naive reference points are chance, which is ~26% on ViewSpatial-Bench and ~29% on PerspectBench as reported in the cited papers, and the camera-frame policy. On MP-Persp the camera-frame policy is chance by construction. On the public benchmarks its accuracy is a measured quantity rather than an assumption.

The strongest end-to-end competitor is MVSM, which reports 82.09% overall accuracy on ViewSpatial-Bench, a 46.24-point absolute gain over base Qwen2.5-VL-3B ViewSpatial-Bench. It is the strongest competitor because it is the only cited method that improves the model itself rather than wrapping it in a pipeline. We will run the released checkpoint, or a re-trained one if it is not released, and also our own reproduction from the same base model with the same 43,000 samples. Our matched-pair model uses an identical sample count, number of steps, and hyperparameter search budget.

Untrained reference models are GPT-4o, reported at 34.98% on ViewSpatial-Bench, and InternVL2.5-8B, reported at 43.24%.

The strongest pipeline competitor is SymPL, which reports 97.33% on COMFORT# "closer" and 91.50% on "facing" questions. We run it on MP-Persp as an upper reference for explicit geometric lifting. For context, TEP reports 66.73% on InterRef. No reported result was found on Lacuna for any method on matched-pixel minimal pairs, so all MP-Persp numbers will be new.

The key ablation trains on the same pairs without pixel matching: the agent is re-rendered with natural lighting and the backgrounds vary between members of a pair. This isolates the contribution of matching from that of minimal-pair structure.

Beating MVSM means three things. Our model must show higher pair consistency on MP-Persp, meaning both members of a pair are correct. It must not exceed chance in the agent-absent condition. And it must fall no more than a few points below MVSM on the unmatched ViewSpatial-Bench.

Why not just…?

Why not just score the camera-frame answer as the only shortcut? It is the dominant shortcut, and if it explained all of MVSM's residual errors and gains, the low-level probes would add nothing. In that case we would report exactly that. However, COMFORT's finding that color and size shift spatial confidence suggests appearance-based shortcuts exist beyond the camera frame. The agent-blind probe measures them directly.

Why not just apply SHINE-style normalization to benchmark images at test time? Whole-image normalization changes everything at once, and the file notes it can distort the structure needed to perceive 3D scenes. It also cannot remove a shortcut that is correlated with the answer across items. We include whole-image normalization as an evaluation arm. We expect it to change accuracy little, and that result would itself indicate that the bias is not a pixel-statistics artifact at inference time.

Why not just use COMFORT's rotation sweeps? COMFORT varies rotation but does not counterbalance the camera-frame answer against the agent-frame answer, and it does not hold background pixels fixed. Its curves can therefore be partly matched by a camera-frame policy. We reuse its rendering setup but change how answers are assigned.

Why not just scale data, as MVSM does, or use symbolic maps? Scaling model size is already ruled out by the flat PerspectBench results across 37 models. Scaling data is the real threat. If MVSM passes our minimal pairs and the agent-absent control, the shortcut hypothesis fails and stage three becomes unnecessary. Symbolic pipelines will likely win on accuracy. We treat them as a ceiling rather than a rival, because the question is what the model computes itself, without roughly 13 seconds per image of external tooling.

Decisive Checks

The hypothesis is supported if three findings hold together. The shortcut policies must reach well above chance on at least one public benchmark. MVSM's advantage must shrink substantially on the shortcut-free subset and on MP-Persp flip pairs. And MVSM, or our unmatched ablation, must stay above chance with the agent removed.

The hypothesis is refuted if MVSM's pair consistency on MP-Persp approaches its public-benchmark accuracy and its agent-absent accuracy is at chance. In that case current fine-tuning is learning the transformation, and the contribution reduces to a cleaner benchmark.

Matched-pair training succeeds only if it beats the unmatched ablation on pair consistency at equal budget.