Do Vision-Language Models Hallucinate an Upright Camera?

When a vision-language model answers a 3D question such as "is the lamp higher than the table?" from a photo taken at an unusual angle, it often gets the answer wrong. 3DSRBench reports accuracy drops of 13.5% for GPT-4o and 32.2% for Gemini-1.5-Pro when moving from common to uncommon camera viewpoints. The usual reading is that perception failed. This note tests a more specific explanation. We propose that many of these errors are hallucinations driven by a canonical-camera prior: the model behaves as if every photo were taken by an upright, eye-level camera. It therefore treats "up in the image" as "up in the world" and "lower in the frame" as "closer". On this view the model may already encode, internally, that the camera is tilted, and simply fail to use that information when it reasons. That would be the perception-reasoning disconnect documented for diagrams and charts, now appearing in geometry.

The distinction matters because it decides what kind of fix can work. If the tilt is never perceived, only better perception will help, whether external depth and pose experts as in DiSR or retraining. If the tilt is perceived but ignored, a cheap inference-time intervention that makes the model condition on its own pose estimate should recover much of the lost accuracy. Such an intervention needs no extra models and no retraining. We propose to build the paired-viewpoint diagnostic that separates these two cases, and to test one such intervention against the simpler alternatives a skeptic would reach for.

Core Claim

The claim is testable in three parts. First, errors on uncommon views are not uniformly spread. They concentrate on items where the "2D-literal" answer differs from the true 3D answer, and when they occur they match that 2D-literal answer far more often than chance. The 2D-literal answer is the one obtained by reading image axes as world axes. Second, the middle-layer hidden states of open vision-language models encode camera pitch and roll, and a linear probe can read them out accurately even on the items the model gets wrong. Third, adding a small activation shift at those layers, scaled by the model's own internally decoded tilt, closes a substantial fraction of the common-to-uncommon accuracy gap. It should do so without harming general hallucination benchmarks.

If the second part fails, so that the tilt is not linearly present, we will report that inference-time steering is the wrong tool for this failure and that the bottleneck sits in the visual encoder. That outcome is also informative.

How This Relates to Prior Work

Three bodies of work motivate the hypothesis.

The reasoning-gap literature shows that models recognize parts but fail to compose them. MATH-Vision traces competition-math failures to logic applied after correct recognition. GeoSQA shows that failures persist even when diagrams are replaced by text. Plot Twist shows that models lean on linguistic shortcuts. We ask whether camera geometry is another percept that is present but unused.

The hallucination literature shows that language priors override weak visual evidence. POPE establishes this for object existence. Villa et al. show models guessing from context, and Yan et al. show models fabricating detail in explanations even when the final answer is right. All of these study 2D co-occurrence priors. We study a geometric prior instead.

The viewpoint literature explains why pose is hard. Li et al. show that variation caused by pose can exceed variation between identities.

Two lines of work supply our method, and we depart from each.

DREAM learns a residual in feature space that maps profile faces toward frontal ones. Its soft gate is driven by a yaw estimate, so frontal inputs pass through untouched. We reuse the gating idea but not the goal. For identity, pose is a nuisance to be erased. For spatial questions, pose is evidence. Mapping uncommon-view features to "canonical" ones would delete the very information the answer needs. Our shift therefore aims to make the pose signal influence the answer, not to remove it.

SHARP and AFTER show that hallucination has linear signatures in middle layers (roughly layers 9–17) and steer along them at inference. Their directions come from 2D object-presence contrasts. Ours comes from a paired-viewpoint contrast and is gated by an internal pose readout. Neither paper does this.

Two papers constrain the design.

Thinking Before Looking plans the reasoning from text alone before looking at the image. Its authors report only about a 3.15% gain on MathVista, because geometric questions cannot be planned without seeing the image. This rules out blind-planning fixes for our setting.

SpatialReasoner argues that 3D perception must precede computation. It trains a model to emit explicit coordinates, but its errors still propagate from mispredicted coordinates under unusual views. Our intervention keeps the perception-first order but avoids serializing coordinates.

VIZOR builds viewpoint-invariant scene graphs, but it requires full 3D meshes. It is therefore not applicable to single images.

Research Program

Paired diagnostic. The first component is a paired-viewpoint diagnostic set. Following 3DSRBench's synthetic protocol on HSSD scenes, we render each scene and object pair from a common eye-level camera and from controlled sweeps of pitch, roll, and camera height. The 3DSRBench question templates for height, location, orientation, and multi-object relations are then instantiated on every render.

Because the renderer gives exact geometry, each item gets both a true 3D answer and a 2D-literal answer. The 2D-literal answer is computed from projected positions under the assumption that the camera is upright. This splits items into conflict items, where the two answers disagree, and agreement items. The idea extends DiSR's "2D-conflict" subsets from depth to camera tilt.

The headline statistic is the prior-matching error share: among wrong answers on conflict items, the fraction equal to the 2D-literal answer. For multiple-choice questions it is compared with the chance rate. For binary questions we compare error rates on conflict versus agreement items at matched tilt. A further check covers the explanations. We parse free-form rationales for stated spatial relations and score each one against ground truth. This tests whether explanations fabricate an upright layout even when the final answer happens to be correct, echoing the hallucination-in-reasoning finding of Zenz et al..

Probing. The second component trains linear probes at each layer of open models (the Qwen2.5-VL and LLaVA families) to decode camera pitch and roll from mean-pooled image-token states. Probes are trained on agreement items only. They are then evaluated separately on conflict items the model answered correctly and incorrectly. "Encoded but ignored" predicts high probe accuracy on the wrong answers. "Not perceived" predicts low accuracy there.

Pose-gated steering. The third component is the intervention. Let $h_\ell$ be the hidden state at the answer position in layer $\ell$. Let $\hat p$ be the tilt read out by the layer-$\ell$ probe from the model's own activations. Let $v_\ell(\hat p)$ be a steering direction for that tilt. We define the direction as the mean activation shift produced by stating the true camera tilt in the prompt, averaged over training renders in the same tilt bin. The edited state is

$$h_\ell' = h_\ell + \alpha\, g(\hat p)\, v_\ell(\hat p),$$

where $\alpha$ is a strength tuned on a validation split. The gate $g$ is a soft function that is near zero for near-canonical tilt, which plays the role of DREAM's yaw gate. In effect, the edit distills "being told the camera pose" into an activation shift. It is triggered by what the model already perceives, so it needs no external estimator and no extra prompt tokens.

Baselines

Do-nothing reference. The unmodified base model with greedy decoding sits on the same paired renders and on the official 3DSRBench splits. Its common-to-uncommon gap is the quantity everything else must shrink.

Scaling reference. We include Qwen2.5-VL-72B, which SpatialReasoner reports at 54.9% mean accuracy on 3DSRBench. This tests whether the gap simply closes with size.

Strongest existing system. DiSR runs frozen SAM, Metric3D, OrientAnything, and a 2D detector, then reasons over serialized 3D evidence with a LoRA-tuned LLM. It reports 67.62% on 3DSRBench, the highest number in our file. SpatialReasoner's end-to-end model reports 60.3%.

We will report DiSR's published number. Where code is available we will also run it on our paired renders broken down by tilt bin, because its expert modules may themselves degrade under extreme pitch. We do not expect a 7B steered model to beat DiSR's overall accuracy. Beating it means closing the viewpoint gap: a smaller common-to-uncommon drop than DiSR at a fraction of its inference cost, while staying within a few points of its common-view accuracy.

Test-time augmentation. RITUAL combines predictions on the original image and on randomly flipped, rotated, or cropped versions during decoding. It is the most direct existing hallucination fix that touches geometry; it reports POPE accuracy for LLaVA-1.5 rising from 84.13% to 88.87%. No reported result was found on Lacuna for RITUAL on 3DSRBench, so we run it ourselves with its published augmentation set on the same base model.

Existing steering. SHARP is applied with its published HaloQuest-derived vectors at its layers. It reports LLaVA-Next adversarial POPE accuracy rising from 79.5% to 82.8%. This isolates whether any hallucination direction helps, or only a viewpoint one.

Ablations. These isolate the key ingredient. We test the same edit without the gate, so it is applied to every input. We test it with a direction from randomly paired renders. We test it with the gate driven by ground-truth tilt instead of the internal probe; this gives an upper bound on what the internal readout loses.

Fairness. All open-model methods share the same base checkpoint, the same evaluation items, and the same validation split for their single strength or temperature parameter. Each method gets the same tuning budget of roughly a few dozen settings.

Why not just…?

Why not just fine-tune on uncommon-viewpoint renders? This is the most serious threat. A LoRA trained on the same paired renders used to build our steering directions might close the gap outright. We will run exactly that, with the same images and the same tuning budget. If it matches the steering on 3DSRBench uncommon splits, the methodological claim shrinks to the diagnosis plus the data. The remaining case for steering would then rest on two measurable points. The first is preservation of general behavior, which we check on POPE, MME, and HaloQuest, where fine-tuning on narrow spatial data may cost accuracy. The second is transfer to the real-image MS-COCO split of 3DSRBench, which neither method sees during training. We commit to reporting this comparison whichever way it goes.

Why not just tell the model the camera pose in the prompt? This is a real alternative, and our steering direction is defined as its distillation. We therefore run it in three variants. The first states the ground-truth pose. The second states the pose from an external single-image estimator. The third has the model first state its own estimate of the tilt and then answer (self-verbalization). If self-verbalization matches the steering, the cheaper and more transparent fix wins, and we will say so. The steering's only advantages would then be fewer generated tokens and no dependence on the model phrasing its estimate correctly. The ground-truth-pose variant separately tells us whether knowing the pose suffices at all. If it does not, both fixes are moot, and the failure lies in geometric computation rather than in the prior.

Why not just rectify the image or use random 2D augmentation? For pure roll, rotating the image upright is an exact fix, and we expect it to work. We include it and report results per tilt type. Pitch and camera height, however, change perspective and occlusion in ways no image-plane warp undoes, the point made by UV-GAN about self-occlusion. RITUAL's augmentations are random rather than pose-directed. 3DSRBench's own FlipEval shows that horizontal flips expose directional bias rather than remove it. The decisive evidence is the per-tilt-type breakdown: rectification should close the roll gap and leave the pitch gap.

Why not just use a bigger model or a modular pipeline? The scaling evidence is already unfavorable: Gemini-1.5-Pro still drops 32.2% on uncommon viewpoints. DiSR is strong, but its authors identify perception under clutter and small objects as its bottleneck. Our tilt-binned evaluation tests whether that bottleneck also covers extreme camera angles.

Decisive Checks

The program succeeds or fails on four measurements, all taken on the paired renders and the official 3DSRBench common and uncommon splits.

The first is the prior-matching error share. A value well above chance on conflict items would establish the canonical-camera hallucination; a value at chance would falsify it and end the steering line.

The second is probe accuracy for tilt on wrongly answered conflict items. High accuracy supports "encoded but ignored"; low accuracy points to perception and to DiSR-style or retraining fixes.

The third is the fraction of the common-to-uncommon gap closed by pose-gated steering, compared with the ungated, random-direction, ground-truth-gate, prompted-pose, rectification, RITUAL, SHARP, and LoRA baselines. Accuracy on POPE adversarial, CHAIR, MME, and HaloQuest is reported alongside, so that gains on 3D questions are not bought with new hallucinations elsewhere.

The fourth is the rate of fabricated upright-layout relations in rationales before and after the intervention. This checks whether the fix changes the reasoning or only the final token.

A clean positive result would show that one geometric failure of vision-language models is a prior-driven hallucination over information the model already holds, and that the model can be made to use that information cheaply. A clean negative result would locate the failure in perception and redirect effort toward better visual encoders.