When a vision-language model (a model that answers questions about images) says "yes, the mug is to the left of the lamp from the girl's point of view" and it is not, the answer can fail in three ways. The model may have misread the picture. It may have read the picture correctly and then failed to rotate the scene into the girl's viewpoint. Or it may have simply agreed with the way the question was phrased. Existing benchmarks cannot tell these apart. Spatial benchmarks use fixed photo sets that are increasingly likely to have leaked into training data. Sycophancy benchmarks — sycophancy meaning the tendency to agree with whatever the user asserts — add leading questions to those same fixed sets.
We propose a procedurally rendered 3D scene generator in which the true spatial layout, the viewpoint difficulty, and the claim embedded in the user's question are set independently. Each scene is queried under a small, fixed set of matched conditions. These separate perception errors, reasoning errors, and agreement with a false premise, all on scenes no model can have seen before. Because we control generation, we can also contaminate a model on purpose. That lets us test a claim no current benchmark can check: memorized items make models look less sycophantic than they are, so sycophancy rates measured on static benchmarks are biased.
Core Claim
Our central empirical claim is twofold. First, affirmation bias on spatial questions is not a constant "yes-rate". It grows with how hard the spatial computation is for the model. In particular, it grows with the angle between the camera and the viewpoint the question refers to. After conditioning on whether the model answered correctly without any leading premise, a residual premise effect remains, and that residual is the part that is genuinely compliance.
Second, memorization shields against that compliance. On items a model has been trained on, false premises flip answers less often than on fresh twins of the same difficulty. Static-benchmark sycophancy estimates therefore mix two things: the model's willingness to agree and the strength of its memorized answer.
What is new here is not "a dynamic spatial benchmark". DynaSolidGeo already re-renders solid-geometry problems under new viewpoints and measures a memorization tax, a 20.4% drop for Claude-Sonnet-4.5. The contribution is a matched-condition design that assigns each failure to perception, transformation, or compliance. It is paired with a controlled-contamination experiment that measures how leakage distorts sycophancy numbers.
What Prior Work Establishes and Leaves Open
Several papers motivate the gap. Spatial-DISE uses a Blender generator for folding and rotation puzzles and attributes 72.5% of failures to reasoning rather than perception. DynaSolidGeo finds the opposite for Gemini-2.5-Pro, where more than half of failures are misreadings of 3D structure. Both rely on after-the-fact error annotation, often by a language-model judge. We replace that annotation with conditions that measure perception and transformation directly.
ViewSpatial-Bench and the PerspectBench study in Vision Language Models See What You Want but not What You See show that models collapse to near chance on perspective-taking and prefer the camera's own viewpoint. Both are static, and neither asks what happens when the user supplies a wrong viewpoint claim. R-Bench shows that models confirm expected spatial relations on illusion images, with most models at 0%. It uses fixed yes/no questions and no leading premises.
On the sycophancy side, the methods we reuse and compare against are these. Towards Analyzing and Mitigating Sycophancy in Large Vision-Language Models adds GPT-4V-written leading queries to POPE and other static sets, and proposes contrastive decoding that subtracts the output distribution of the leading query from that of a neutralized query. Pointing to a Llama and Call it a Camel shows that models are more sycophantic with images than with text descriptions of the same images. Its reflective-tuning method, SRT, preserves correctability, meaning acceptance of legitimate user corrections, where plain fine-tuning destroys it. System-Mediated Attention Imbalances Make Vision-Language Models Say Yes traces the yes-bias to late-layer attention being absorbed by system-prompt tokens, and removes part of it by redistributing that attention.
All three measure sycophancy on public, fixed images. None separates compliance from spatial incompetence. HallusionBench comes closest to our contamination question: its human-edited control images catch models answering from memorized knowledge instead of the picture. It is small, static, and not spatial in a controlled sense.
The contamination literature constrains the design. Investigating Data Contamination in Modern Benchmarks shows that surface changes do not remove memorization; GPT-4 guesses masked MMLU options at 57% exact match. BeyondBench and DyVal show that parametric regeneration exposes collapses that static sets hide. Both are text-only, which is why we regenerate the scene itself rather than paraphrase the question.
Research Program
The generator renders tabletop scenes of two to five objects. It uses a mix of primitives and household-object meshes chosen to echo the What'sUp setting of What's "up" with vision-language models?. The scene may also contain a person or doll avatar with a known facing direction. From the scene graph we compute ground truth for left/right, in front/behind, on/under, nearer the camera, and left/right from the avatar's viewpoint.
The main difficulty knob is the viewpoint offset $\theta$, the angle between the camera and the frame of reference the question uses. It runs from 0° (the camera's own frame) to 180° (facing the camera). Secondary knobs are occlusion, object count, and distractor similarity. Every item is freshly sampled at evaluation time, so the chance of overlap with training data is negligible, as in BeyondBench.
Each sampled scene is queried under six matched conditions. The first is a camera-frame read-out that needs no transformation and serves as the perception probe. The second is a text-only condition: an exact textual description of object coordinates and avatar pose, which removes perception and isolates the transformation step. The third is a neutral image question. The fourth embeds a true premise in the question, and the fifth a false premise. The sixth, a correction condition, applies only after a wrong neutral answer: the user supplies the right answer, which measures correctability. Premise polarity is balanced, and the true answer is "yes" half the time, so a model that always agrees gains nothing.
Failures are then assigned as follows. A perception failure is an error on the camera-frame probe. A transformation failure has a correct probe and correct text-only reasoning, but a wrong neutral image answer at $\theta>0$. Where text-only reasoning fails too, the failure is a pure reasoning deficit. Compliance is defined only on items the model answers correctly when neutral:
$$C(\theta) = P\big(\hat{y}_{\text{false}} = p \mid \hat{y}_{\text{neutral}} = y,\ \theta\big)$$
Here $y$ is the true answer, $p$ is the false claim in the premise, and $\hat{y}_{\text{neutral}}$ and $\hat{y}_{\text{false}}$ are the model's answers without a premise and with the false premise. Correctability $R(\theta)$ is the analogous probability of adopting a true correction after a wrong neutral answer. The hypothesis that compliance depends on difficulty is that $C(\theta)$ rises with $\theta$ even after conditioning on the model's neutral-answer confidence, taken as the token probability for open models and the agreement rate over sampled answers for closed ones.
The contamination arm is controlled. We fine-tune Qwen2.5-VL-7B with low-rank adapters on 2,000 rendered items in neutral question-answer format for several epochs, which simulates leakage. We then compare $C(\theta)$ on those seen items against fresh twins of identical difficulty. The prediction is that seen items show lower compliance, meaning memorization shields the model from the premise. The size of that gap, the "sycophancy masking" effect, is the correction that static sycophancy numbers need.
Finally, we build a real-image arm: leading-premise versions of What'sUp and ViewSpatial-Bench items, plus rendered twins that match their layouts. This measures how much of any gap between rendered and real results reflects the generator's synthetic look rather than contamination.
Baselines
The naive reference point is the evaluation the field already runs. We take static spatial sets (What'sUp, VSR from Visual Spatial Reasoning, ViewSpatial-Bench), add leading queries in the manner of Towards Analyzing and Mitigating Sycophancy, and report the accuracy drop. On POPE that paper reports that leading queries cause accuracy drops of up to 42% and F1 falling to 10%. For spatial leading queries we found no reported result on Lacuna.
We run this static arm on exactly the same models and premise templates as the rendered arm. Our design beats it if the static arm's compliance estimates differ from the rendered estimates by more than the gap explained by synthetic appearance, as measured through the rendered twins. It also beats it if the static arm cannot reproduce the rise of compliance with $\theta$.
The strongest existing mitigation, and therefore the method whose effect our diagnosis must explain, is the contrastive decoding method ITSM from the same paper, which restores POPE F1 "to the 80-90% range". SRT from Pointing to a Llama is the correctability-preserving competitor. It reduces Qwen2-VL-7B's sycophancy rate from 13.00% to 3.47% while keeping a correction rate of 28.86%, against 6.18% for plain fine-tuning. We also include late-layer system-attention redistribution from System-Mediated Attention Imbalances, which lowers LLaVA-1.5's HallusionBench yes-rate from 89.8% to 59.6%. Finally, we include the spatial-attention fix AGCI from From Bias to Balance as a perception-side intervention.
All mitigations are applied to the same open models (LLaVA-1.5 7B, Qwen2.5-VL-7B, InternVL2.5-8B) on the same sampled scenes, with each method's published hyperparameters and an equal small tuning budget on a separate held-out batch of scenes. For the sake of the diagnosis, our success criterion is not beating these methods. It is predicting their effects. The decomposition should say that contrastive decoding lowers $C(\theta)$ without changing perception or transformation errors, while AGCI moves only the perception probe. If the mitigations instead move all components together, the decomposition is not separating real causes.
Two ablations isolate the key ingredients. Dropping the text-only condition merges transformation failures with perception failures, and shows whether separating those two steps changes conclusions such as the Spatial-DISE versus DynaSolidGeo disagreement. Dropping the conditioning on neutral correctness turns $C$ back into a raw yes-rate. We then test whether its dependence on $\theta$ disappears or reverses.
Why not just…?
Why not just add leading questions to existing spatial benchmarks? This is the cheapest option and a real threat. If memorization does not interact with compliance, a static arm would give the same compliance curves at a fraction of the cost. The controlled-contamination experiment settles it directly. If seen and fresh items show indistinguishable $C(\theta)$, we will say static augmentation is adequate for compliance measurement. The generator would then be justified only by its viewpoint knob and by resistance to future leakage. The evidence that leakage matters for spatial items is indirect: the 20.4% memorization tax in DynaSolidGeo and the language-prior override in HallusionBench.
Why not just report yes-rate or balanced accuracy? A yes-rate treats "agrees because it cannot compute the answer" the same as "agrees despite computing it". Balanced premise polarity and conditioning on correct neutral answers are exactly what separate the two. The raw-yes-rate ablation shows whether this changes any ranking of models or mitigations. If rankings stay the same, the extra conditions are not worth their cost, and we will report that.
Why not just treat sycophancy as low confidence? A skeptic may argue that $C(\theta)$ rises with $\theta$ only because the model is less sure at larger offsets. That would make "affirmation bias" reducible to calibration. We test this by stratifying on neutral-answer confidence. If the premise effect vanishes within confidence bins, the honest conclusion is that spatial sycophancy is uncertainty plus a prior toward agreement. That result would still be useful, because it points to calibration fixes rather than anti-sycophancy training. The finding in Pointing to a Llama that the same content is more sycophancy-prone as an image than as text suggests a residual beyond confidence. Our text-only condition tests that residual directly.
Why not just use Spatial-DISE's or DynaSolidGeo's generators? Their tasks, folding and solid-geometry calculation, have no natural frame of reference that the user could misstate. Their outputs are also released as fixed sets or judged by a language model. We reuse their approach of rendering and scripted variation, but we need live sampling, a viewpoint knob, and exact scene descriptions for the text-only condition, which those generators do not produce.
Decisive Checks
The first check is whether $C(\theta)$, conditioned on correct neutral answers and stratified by confidence, has a positive slope in $\theta$ for at least three of the five models. The five are the three open models plus GPT-4o and Gemini, evaluated on about 5,000 fresh scenes per model. The second is whether the controlled-contamination gap in compliance between seen and fresh items is significantly non-zero. The third is whether the three mitigation families move the component they should move and leave the others unchanged.
A fourth check tests the decomposition at its most interesting point. For spatially trained models, MVSM from ViewSpatial-Bench and SpatialThinker, the decomposition predicts that compliance falls only to the extent that transformation failures fall. If spatial training reduces compliance on items the models already answered correctly, then spatial competence and resistance to agreement are more entangled than our model assumes, and we will say so.
References
- DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry
- Spatial-DISE: A Unified Benchmark for Evaluating Spatial Reasoning in Vision-Language Models
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Vision Language Models See What You Want but not What You See
- Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models
- Towards Analyzing and Mitigating Sycophancy in Large Vision-Language Models
- Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models
- System-Mediated Attention Imbalances Make Vision-Language Models Say Yes
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- Investigating Data Contamination in Modern Benchmarks for Large Language Models
- BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Visual Spatial Reasoning
- From Bias to Balance: Exploring and Mitigating Spatial Bias in LVLMs
- SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards