TL;DR
The CueClash benchmark reveals that frontier Vision-Language Models (VLMs) like GPT-5.5 and Claude Opus 5.5 often rely on 2D image shortcuts rather than true 3D scene reconstruction. While these models can reason perfectly when given 3D coordinates, they fail to accurately recover layouts from images, consistently "compressing" depth toward the 2D picture plane in a manner reminiscent of human "phenomenal regression."
What Problem Does This Paper Address?
Vision-Language Models are increasingly used for spatial reasoning, but it’s often unclear if they are truly understanding the 3D scene or just exploiting 2D shortcuts. For example, in a natural photograph, an object that is higher in the frame is usually further away. A model might learn this "shortcut" without ever understanding depth.
Existing benchmarks struggle to separate these because 2D cues and 3D truth usually align in real-world photos. The authors address this by creating a controlled environment where 2D cues are intentionally set against 3D truth, forcing models to choose between the "picture plane" (the 2D image) and the "ground plane" (the actual 3D layout).
Methods
The authors introduce CueClash, a benchmark of rendered scenes containing 4 to 8 geometric primitives. For each of five spatial relations—closer, taller, touching, between, and fits—they identify a common 2D shortcut (e.g., comparing pixel widths to judge if an object "fits" through a gap).
Key methodological innovations include:
- Balanced Cue Conflict: Scenes are designed so that the 2D shortcut is correct exactly 50% of the time.
- Shortcut Gap: A new metric defined as the model's accuracy when the 2D cue is right minus its accuracy when the cue is wrong. A high gap indicates heavy reliance on 2D shortcuts.
- Input Form Ablation: Models are tested with different inputs: the raw image, pixel measurements, camera depths, and ground coordinates. This helps pinpoint whether the failure is in perception (extracting the layout) or reasoning (applying the spatial rule).
- Drawn Maps: Models are asked to "draw" (output JSON coordinates) a bird's-eye view of the scene to visualize their internal representation of the 3D layout.
Datasets
- CueClash (New): 1,800 rendered items across three difficulty tiers (easy, medium, hard).
- CV-Bench: Used to validate findings on real-world photographs where 2D cues occasionally mislead.
Key Insights and Ideas
- Reasoning vs. Perception: Models reason nearly perfectly (99-100%) when given ground coordinates. Their failure to answer correctly from images is a failure of perception—specifically, the inability to accurately recover 3D layouts from 2D pixels.
- Depth Compression: When models "draw" the scene from above, they consistently shorten the depth dimension relative to width. This "depth compression" makes the recovered layout look more like the 2D picture than the 3D scene.
- Phenomenal Regression: Claude Opus 5.5 shows a "constancy index" of ~0.6, meaning it corrects for about 60% of the 2D distortion. This is remarkably similar to human observers. In contrast, GPT-5.5’s internal maps stay very close to a literal copy of the 2D picture (index ~0).
- Hidden State vs. Final Answer: Probing open models (like Qwen3-VL) reveals that 3D layout information is present in their internal hidden states, but the language model fails to utilize it for the final "True/False" answer, opting for the 2D shortcut instead.
Key Results
- Large Shortcut Gaps: At low reasoning effort, frontier models show shortcut gaps of +18 to +37 percentage points. GPT-5.5 performed below chance on "fits" items where the 2D cue was wrong.
- Fits and Between are Hardest: The gaps are most pronounced for relations that require comparing sizes across depth (fits) or judging alignment (between).
- Real-World Impact: The same models lose significant accuracy on CV-Bench photographs in the rare cases where 2D cues (like "lower in frame = closer") are misleading.
- High Reasoning Effort: Increasing the reasoning budget (CoT, more tokens) narrows the gap for some models but does not eliminate it, suggesting the perceptual bottleneck is fundamental.
Limitations and Future Work
- Synthetic Primitives: The benchmark uses simple geometric shapes on a flat plane. While this allows for exact balancing, it lacks the complexity of natural textures and lighting.
- Human Baselines: The paper lacks direct human performance data on the CueClash items, relying instead on historical literature for human depth constancy values.
- Relation Confounding: For "touching" and "between," the authors note that cue-wrong items are sometimes confounded by occlusion or borderline cases, making the "shortcut gap" harder to interpret for these specific relations.
- Future Direction: The authors suggest that VLM development should specifically target the "pixels-to-layout" step, perhaps by integrating explicit depth sensing or map-based representations into the model's reasoning chain.
Summary updated