Vision-language models, the systems that answer questions about images, can describe a medical scan or guess a person's intent. Yet they often score near chance when asked whether a mug is to the left of a bowl, or what a scene looks like from another viewpoint. The usual fix today is to hand-design an intermediate description of the scene and train the model to write it before answering. That description might be a text scene graph, a JSON list of orthographic views, or a set of discretized direction labels. Building these descriptions takes detectors, depth estimators, 3D scans, or synthetic engines. This is the same position NLP was in when sentiment and coherence systems relied on hand-built parse trees and entity grids. NLP escaped largely by finding training signals that came from raw text. The clearest example is the coherence model of A Model of Coherence Based on Distributed Sentence Representation, which learned discourse structure by telling real paragraphs apart from paragraphs with shuffled sentences, using no parser and no coreference features.
We propose the visual analogue. We take an ordinary unlabeled photo and edit it so that its spatial arrangement changes while everything else stays the same: two objects swap places, one object is mirrored, or an object is lifted off its support. The model is then asked which of two images is the unedited one, and it is rewarded for answering correctly. The right answer is known from the edit itself, so this is a reward the computer can check without any human spatial labels, scene graphs, or 3D scans. The claim we test is that reinforcement learning on this label-free coherence task alone moves a 3B-parameter vision-language model toward the spatial accuracy of methods trained on curated spatial question-answer data. We further claim it generalizes better to controlled tests, because the only way to solve the task is to read geometric cues such as contact, occlusion order, shadows, and size changes with depth.
Core Claim and How It Differs From Existing Work
The obvious idea at this intersection is to replace hand-built text intermediates with learned continuous "latent thoughts," and that idea is already taken. LatentGeo generates continuous latent tokens in place of drawn auxiliary lines for geometry problems. LanteRn lets a model interleave text with latent visual vectors. Spa3R learns a view-invariant scene code by predicting features of unseen views and plugs it into Qwen2.5-VL. All three change the representation, and all three still depend on curated supervision: auxiliary-diagram ground truth, Visual-CoT bounding boxes, or posed ScanNet scans.
Our proposal leaves the architecture alone and attacks the other half of the feature-engineering bottleneck, which is where the supervision comes from. The closest relative is Spa3R, which also uses self-supervision. Spa3R, however, needs calibrated multi-view indoor scans and a separate frozen encoder. We use single unposed images and short unposed video clips, and we train the whole model through its normal answer channel.
The explicit-intermediate methods motivate the work by showing that structure helps, and they also serve as baselines. SpatialThinker trains Qwen2.5-VL-7B with reinforcement learning to emit a scene graph with boxes before answering, using about 7,600 curated synthetic questions. Scene Graph Thinking (SaGe) builds 120,000 graph-annotated samples with detectors, segmenters, and depth models. 3ViewSense has the model write front, top, and side views as JSON. ViewSpatial-Bench's MVSM turns 3D orientation into 18 direction labels using ScanNet and an orientation model. ReCLIP is the extreme case: it passes a spaCy parse to hard-coded 2D box rules.
Two papers constrain the design. The finding in What's "up" with vision-language models? that spatial prepositions appear in under 0.2% of LAION captions rules out mining spatial supervision from captions at scale. The same paper's observation that fine-tuning on relational question-answer data hurt on its controlled What'sUp set warns that narrow supervised spatial data invites shortcuts. A coherence objective is the kind of signal that might avoid this, but only if the edits are built so that low-level artifacts cannot give the answer away.
One finding cuts against an encoder-only version of this idea. 3ViewSense reports that a linear probe on frozen visual features already reaches 55.8% on block counting, while the full model gets 6.2%. Spatial information is present in the image features but unused by the language side. We therefore pose the coherence task to the full model as a question it must answer in text, so the reward reaches the part that reads the visual tokens, not only the vision encoder.
Research Program
The core artifact is a generator of matched image pairs. In every pair, both images pass through identical editing machinery, and they differ only in geometry.
For the object-swap family, a class-agnostic mask generator proposes two object regions. In the negative image, the two regions are cut, swapped, pasted, and the holes are inpainted. In the positive image, the same regions are cut and pasted back in their original positions with the same inpainting and blending. Seams, blur, and inpainting texture are therefore present in both images, and the only difference is where the objects sit. The mask generator outputs no labels, relations, or depth, which is a far lighter dependency than SaGe's full graph pipeline. We also run a variant with plain rectangular patches and no segmenter, to measure how much even this dependency matters.
The mirror family flips a single object region horizontally in place. It is matched against an identical re-paste of the unflipped region, and near-symmetric objects are filtered out by pixel difference.
The support family moves an object vertically off its resting surface, matched against a re-paste at the same height.
The view-order family uses unposed video. Three frames are taken from a camera trajectory, and the model must tell the true order from an order where two frames are swapped. This is the closest literal copy of Li and Hovy's sentence-permutation task.
Training runs Group Relative Policy Optimization (GRPO), the reinforcement-learning method used by SpatialThinker and 3ViewSense, which scores several sampled answers per prompt against each other. We start from Qwen2.5-VL-3B, the backbone of MVSM, SaGe-3B, and LanteRn. The reward is answer correctness plus a format term. There is no scene-graph tag and no box reward.
Many swapped arrangements are perfectly plausible: a cup can sit on either side of a plate. These pairs are label noise for the coherence task. We keep them in the main run and measure their effect by separately training on a subset that a small held-out discriminator finds decidable. Our hypothesis is that the decidable cases teach the cues that matter, namely contact, occlusion ordering, shadow direction, and perspective scale.
The analysis side asks what was learned. We run linear probes for 2D centroid order and relative depth on the model's visual tokens before and after training. We also run attention attribution on What'sUp items, checking whether answers depend on the two named objects or on the global scene.
Baselines
The naive reference point is the untouched Qwen2.5-VL-3B, prompted zero-shot on every benchmark with the same answer templates. ViewSpatial-Bench's authors report this model at 35.85% on their benchmark, derived from MVSM's reported 46.24-point gain.
The strongest existing method for allocentric direction questions is MVSM-3B from ViewSpatial-Bench. It reports 82.09% overall accuracy on ViewSpatial-Bench using 43,000 curated multi-perspective samples built from ScanNet and an orientation model. We report it as published. We do not expect to beat it on its own benchmark without 3D data. The relevant comparison is how much of its gain we recover with zero spatial labels, and whether we exceed it on What'sUp and VSR, which it was not built for.
The strongest competitor on our actual question is an explicit scene-graph reinforcement-learning recipe, because it shares our optimizer and differs exactly in where supervision comes from. We will reimplement the SpatialThinker recipe from SpatialThinker on Qwen2.5-VL-3B with its STVQA-7K data and lexicographically gated rewards for format, count, accuracy, and box overlap. SpatialThinker-7B reports 76.3% on RoboSpatial-Home and 65.9% on MM-Star. On 3DSRBench it reports only a 12.1-point margin over GPT-4o, so no absolute 3DSRBench number for it was found on Lacuna. For fairness, both arms get the same GRPO steps, group size, rollout token budget, learning-rate sweep of equal size, and checkpoint selection on the same held-out spatial validation split.
SaGe is a further published reference on CVBench, where SaGe-7B reports 79.4 (2D) and 80.5 (3D). For SaGe-3B it reports only gains of 10.8 and 5.8 points over its base. We compare against these published numbers with that caveat, not against a reimplementation.
Two ablations isolate the key ingredient. In the first, the edits change appearance but not geometry: an object's color is shifted, or it is replaced by a different object at the same location, again with matched artifacts. If this arm helps spatial benchmarks as much as ours does, then the gain comes from generic reinforcement learning on image-comparison tasks, not from spatial coherence. In the second, the artifact matching is removed so that only negatives are edited. This tests whether the model learns seam detection instead of geometry.
Our method counts as beating the scene-graph recipe if it is equal or better on What'sUp, VSR, and CVBench at equal compute, or if it is within two points in-distribution while being better on the controlled What'sUp set.
Why not just…?
Why not just scale the model or the web data? Vision Language Models See What You Want but not What You See tested 37 models. Scale improved intention inference, but no model beat chance (about 29%) on PerspectBench, and the two abilities correlated at r = 0.0252. GPT-4o sits at 34.98% on ViewSpatial-Bench. The caption-statistics result explains why: the web rarely states where things are. We include GPT-4o's published numbers as a scale reference and do not pursue this alternative further.
Why not just add hard-negative captions with swapped prepositions to contrastive or instruction training? This is a real threat, because it is cheap and targets the same confusion. Its supply is limited by how rarely spatial prepositions appear in captions. The What'sUp paper's finding that relational fine-tuning degraded on its controlled set suggests that caption-level spatial supervision overfits to dataset phrasing. We will settle this empirically with a third arm. That arm runs GRPO on caption-swap questions generated from COCO captions that contain spatial words, using the same compute. Our method must beat it on What'sUp to justify the extra machinery.
Why not just keep using explicit scene graphs, since they work? In-distribution they do work, and we may not beat them there. The reason to look past them is cost and transfer: they need curated or tool-built spatial labels, and their intermediate text discards metric detail, as Position: Reasoning After Perception Means Reasoning Without Vision argues. The matched comparison above decides whether label-free coherence is a substitute. We also test whether it is a complement by running coherence training followed by the scene-graph recipe, and checking whether the combination beats the scene-graph recipe alone.
Why not just train a better vision encoder, as Spa3R does? The 3ViewSense probe result suggests the encoder already carries much of the signal. We will compare against freezing the language model and training only the encoder on the coherence task. Our prediction is that this variant gains less, which would confirm that the bottleneck is in reading the visual features, not producing them.
Proposed Experiments
All arms are trained from Qwen2.5-VL-3B. The coherence arms use COCO training images with their annotations discarded, plus unposed clips from public video for the view-order family.
Evaluation covers What'sUp, where the best reported model, BLIP fine-tuned on VQAv2, reaches 56% against 99% for humans. It covers VSR's random and concept zero-shot splits, where models plateau around 70% and zero-shot CLIP reaches 56%. It also includes CVBench-2D and 3D, ViewSpatial-Bench, VSI-Bench (Spa3-VLM reports 58.6%), and PerspectBench, with MM-Star to check that general ability is retained. Metric is accuracy throughout, averaged over three seeds with bootstrap intervals.
We expect gains concentrated on What'sUp, VSR, and CVBench-2D, where 2D arrangement and contact matter. We expect smaller gains on CVBench-3D and VSI-Bench. We expect little or none on PerspectBench and ViewSpatial-Bench's allocentric items, because no edit family asks the model to adopt another observer's viewpoint. A large gain there would surprise us and would itself need explaining.
Three outcomes would be decisive. First, if the geometry-free edit arm matches the full method on spatial benchmarks, the coherence hypothesis is false. Second, if the unmatched-artifact arm trains to high coherence accuracy but transfers nothing, while the matched arm transfers, then artifact control is the essential design choice. Third, if per-family training shows that downstream gains on "on/under" items track the support family and gains on "left/right" items track the swap family, then the model is learning the specific geometric relation each edit exposes and not a generic edit detector. That is the result that would justify treating spatial coherence the way NLP came to treat sentence order: as a structure the raw data supervises for free.
References
- A Model of Coherence Based on Distributed Sentence Representation
- LatentGeo: Learnable Auxiliary Constructions in Latent Space for Multimodal Geometric Reasoning
- LanteRn: Latent Visual Structured Reasoning
- Spa3R: Predictive Spatial Field Modeling for 3D Visual Reasoning
- SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
- 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Vision Language Models See What You Want but not What You See
- Position: Reasoning After Perception Means Reasoning Without Vision