Sophon: A Procedural Diagnostic for Spatial Reasoning in Vision-Language Models

J. Zach Gazak, Ryan Swindle, Justin FletcherarXiv 2026 · arXiv

TL;DR

Sophon is a new procedural benchmark that reveals a "readout-routing bottleneck" in open-weight Vision-Language Models (VLMs). While these models possess the necessary spatial information in their internal representations, they fail to route it to the final answer token—a flaw that can be repaired by targeted fine-tuning of the language tower's mid-stack "workspace."

What Problem Does This Paper Address?

Modern VLMs are surprisingly good at naturalistic spatial benchmarks like BLINK, often reaching near-human performance. However, the authors argue these scores are inflated by "semantic scaffolding." If a model sees a cup and a table, it can use linguistic priors to guess that the cup is "on" the table without actually performing geometric reasoning.

When these semantic priors are stripped away—leaving only abstract geometric primitives—open-weight models (like LLaVA and Qwen) collapse to chance-level performance. The authors set out to determine if this is a "capability gap" (the model can't see or understand geometry) or a "routing failure" (the model has the info but can't get it to the output).

Methods

The authors introduce Sophon, a procedurally generated diagnostic tool. Unlike natural images, Sophon uses paired, colored point clouds that differ by a single geometric transformation (translation, rotation, or scale). These are presented across three coordinate systems (1D linear, 2D Cartesian, and 2D polar).

To diagnose the failure, the researchers employed several surgical techniques:

  • Linear Probing: They trained simple classifiers on the frozen internal layers of both the vision encoder and the language decoder to see if spatial information was present.
  • Component Ablation via LoRA: They used Low-Rank Adaptation to fine-tune specific "bands" of the model—the vision tower, the projector, or specific layers of the language tower—to see which parts were responsible for the performance gain.
  • Logit Lens: This technique projects the hidden state of every layer into the model's vocabulary space, allowing researchers to "see" when the model starts to favor the correct answer.
Figure 2: Sample Sophon stimuli, one per (question type, coordinate system) cell: 1D linear / categorical (left), Cartesian / binary (center), polar / quantitative (right). Each panel shows its coordinate system and ground truth transform parameter. No objects, scenes, or textures appear in any stimulus (Appendix B).

Datasets

  • Sophon (New): A benchmark of 15,000 procedurally generated instances. It includes four question types: binary (are they aligned?), categorical (which direction?), comparative (which is better?), and quantitative (by how much?).
  • External Benchmarks: The study uses BLINK Spatial and What’s Up CLEVR as baselines to show how models that succeed on naturalistic tasks fail on abstract ones.

Key Insights and Ideas

  • The Information is Already There: Probes of frozen vision encoders (like CLIP or SigLIP) recovered Sophon spatial information at ~85% accuracy. Even more strikingly, probes of the language decoder showed the signal survives intact all the way to the final layer. The model "knows" the answer internally but fails to select the correct token at the very end.
  • The Routing Bottleneck: The failure isn't in perception or reasoning, but in "routing." The authors describe a "workspace" in the mid-stack layers of the language tower that must organize the visual signal so the final "reader" layers can decode it.
  • Workspace Tuning is the Lever: Fine-tuning just the mid-stack "workspace" layers recovered nearly 100% of the performance gain of full-model fine-tuning. Conversely, tuning the final "reader" layers alone provided almost no benefit.
  • Text-Only Superiority: When the point cloud coordinates were fed to the model as text rather than an image, accuracy jumped by 40 percentage points. This confirms the language decoder is perfectly capable of the math; it just struggles to "read" the visual input.
Figure 1: The diagnostic gap and its mechanism. (a) Despite 85−94% on existing spatial bench- marks, open models show no spatial competence on Sophon while untrained humans solve with high accuracy. (b) Ablation localizes the fine tuning mechanism upstream: the reader-band discriminative contrast at the readout is a downstream effect of upstream workspace change (§5.3).

Key Results

  • The Gap: Open 7B-class VLMs clustered at 47–52% accuracy (essentially chance), while frontier models (GPT-5.4, Gemini 2.5 Pro) reached 77–83%, and humans achieved 84–94%.
  • The Fix: Targeted LoRA fine-tuning on just 1,000 Sophon samples allowed 7B models to reach frontier-level aggregate accuracy.
  • Shortcut Failures: The authors identified "constant token" failures, where a model like Grok-4.20 or Qwen2.5-VL would simply repeat the same answer (e.g., "counterclockwise") for every single rotation question, regardless of the image.

Limitations and Future Work

The authors acknowledge that Sophon’s abstract nature means results might not perfectly translate to naturalistic "real-world" competence. Furthermore, the study focused primarily on 7B-parameter models and specific projector architectures (MLP, pixel-shuffle).

A major remaining mystery is rotation. Even frontier models and fine-tuned open models struggle with rotation compared to translation and scale. The authors suggest that the lack of explicit rotation supervision in standard pretraining mixtures might be a fundamental hurdle that simple fine-tuning can't fully clear. Future work could explore whether different visual encoders or "ordered" point cloud representations might bridge this final gap.

Summary updated

Abstract

Open-weight Vision-Language Models (VLMs) score 85-94% on spatial benchmarks (BLINK Spatial, What's Up CLEVR) yet collapse to chance on simple geometric questions when object, scene, and texture priors are stripped away. We introduce and open-source Sophon, a procedurally generated diagnostic of paired colored point-cloud stimuli across three coordinate systems (1D linear, 2D Cartesian, 2D polar) and four question types (binary, categorical, comparative, quantitative). Across four open architectures (LLaVA-NeXT, LLaVA-OneVision, Qwen2.5-VL, Idefics3), accuracy clusters at 47-52% while frontier models (Opus 4.6, Gemini 2.5 Pro, GPT-5.4) clear 77-83% and humans reach 84-94%. We use linear probes and low-rank adaptation (LoRA) to trace the open-weight failures. Probes of the vision encoder (ViT) and language decoder (LLM) recover Sophon spatial information at 83-85% and 80-86% accuracy, respectively, and the language tower's reasoning circuit answers text-only controls (point clouds as sets of coordinates) at +40 pp over visual stimuli. Language tower per-layer attention attribution localizes visual integration to mid-stack layers. At readout, models elevate the correct answer-set tokens but select among them randomly. We localize the mechanism by tuning individual and grouped components across every architecture's stack, including the vision tower (ViT workspace and reader), the projector, and the language tower (LLM workspace and reader). These experiments demonstrate that this mid-stack workspace holds circuitry capable of organizing the spatial signal so that the frozen reader can decode it. LoRA fine-tuning adjusts that circuitry such that 7B-class VLMs answer Sophon spatial reasoning questions at frontier aggregate accuracy. This work suggests that these models do not have a capability gap for spatial reasoning but suffer from a routing bottleneck that targeted fine-tuning repairs.
BibTeX
@misc{gazak2026sophon,
  author = {J. Zach Gazak and Ryan Swindle and Justin Fletcher},
  title = {{Sophon: A Procedural Diagnostic for Spatial Reasoning in Vision-Language Models}},
  year = {2026},
  eprint = {2611.14838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14838},
}
Figure