TL;DR
Sophon is a new procedural benchmark that reveals a "readout-routing bottleneck" in open-weight Vision-Language Models (VLMs). While these models possess the necessary spatial information in their internal representations, they fail to route it to the final answer token—a flaw that can be repaired by targeted fine-tuning of the language tower's mid-stack "workspace."
What Problem Does This Paper Address?
Modern VLMs are surprisingly good at naturalistic spatial benchmarks like BLINK, often reaching near-human performance. However, the authors argue these scores are inflated by "semantic scaffolding." If a model sees a cup and a table, it can use linguistic priors to guess that the cup is "on" the table without actually performing geometric reasoning.
When these semantic priors are stripped away—leaving only abstract geometric primitives—open-weight models (like LLaVA and Qwen) collapse to chance-level performance. The authors set out to determine if this is a "capability gap" (the model can't see or understand geometry) or a "routing failure" (the model has the info but can't get it to the output).
Methods
The authors introduce Sophon, a procedurally generated diagnostic tool. Unlike natural images, Sophon uses paired, colored point clouds that differ by a single geometric transformation (translation, rotation, or scale). These are presented across three coordinate systems (1D linear, 2D Cartesian, and 2D polar).
To diagnose the failure, the researchers employed several surgical techniques:
- Linear Probing: They trained simple classifiers on the frozen internal layers of both the vision encoder and the language decoder to see if spatial information was present.
- Component Ablation via LoRA: They used Low-Rank Adaptation to fine-tune specific "bands" of the model—the vision tower, the projector, or specific layers of the language tower—to see which parts were responsible for the performance gain.
- Logit Lens: This technique projects the hidden state of every layer into the model's vocabulary space, allowing researchers to "see" when the model starts to favor the correct answer.
Datasets
- Sophon (New): A benchmark of 15,000 procedurally generated instances. It includes four question types: binary (are they aligned?), categorical (which direction?), comparative (which is better?), and quantitative (by how much?).
- External Benchmarks: The study uses BLINK Spatial and What’s Up CLEVR as baselines to show how models that succeed on naturalistic tasks fail on abstract ones.
Key Insights and Ideas
- The Information is Already There: Probes of frozen vision encoders (like CLIP or SigLIP) recovered Sophon spatial information at ~85% accuracy. Even more strikingly, probes of the language decoder showed the signal survives intact all the way to the final layer. The model "knows" the answer internally but fails to select the correct token at the very end.
- The Routing Bottleneck: The failure isn't in perception or reasoning, but in "routing." The authors describe a "workspace" in the mid-stack layers of the language tower that must organize the visual signal so the final "reader" layers can decode it.
- Workspace Tuning is the Lever: Fine-tuning just the mid-stack "workspace" layers recovered nearly 100% of the performance gain of full-model fine-tuning. Conversely, tuning the final "reader" layers alone provided almost no benefit.
- Text-Only Superiority: When the point cloud coordinates were fed to the model as text rather than an image, accuracy jumped by 40 percentage points. This confirms the language decoder is perfectly capable of the math; it just struggles to "read" the visual input.
Key Results
- The Gap: Open 7B-class VLMs clustered at 47–52% accuracy (essentially chance), while frontier models (GPT-5.4, Gemini 2.5 Pro) reached 77–83%, and humans achieved 84–94%.
- The Fix: Targeted LoRA fine-tuning on just 1,000 Sophon samples allowed 7B models to reach frontier-level aggregate accuracy.
- Shortcut Failures: The authors identified "constant token" failures, where a model like Grok-4.20 or Qwen2.5-VL would simply repeat the same answer (e.g., "counterclockwise") for every single rotation question, regardless of the image.
Limitations and Future Work
The authors acknowledge that Sophon’s abstract nature means results might not perfectly translate to naturalistic "real-world" competence. Furthermore, the study focused primarily on 7B-parameter models and specific projector architectures (MLP, pixel-shuffle).
A major remaining mystery is rotation. Even frontier models and fine-tuned open models struggle with rotation compared to translation and scale. The authors suggest that the lack of explicit rotation supervision in standard pretraining mixtures might be a fundamental hurdle that simple fine-tuning can't fully clear. Future work could explore whether different visual encoders or "ordered" point cloud representations might bridge this final gap.
Summary updated