Small vision-language models, which answer questions about images, are poor at basic spatial questions such as "is the mug left of or right of the plate?" Fine-tuning them on spatial data often teaches new dataset-specific shortcuts rather than geometry. A separate literature on selective rationalization has studied a closely related failure in text. In that setup, one module picks out a few words as evidence and a second module must predict the label from those words alone. The two modules tend to collude: they invent a private code, for example "a selected period means positive," that scores well but explains nothing. We propose to bring that lens into spatial reasoning. A compact student model is forced to answer only from a small, checkable geometric summary of the scene: boxes, coarse depth, and facing direction for the objects in the question. We then measure and suppress the ways the student hides the answer inside that summary. The same geometric check also serves as a cheap filter for synthetic training data. It replaces the expensive language-model verifiers that current spatial distillation pipelines rely on.
The opportunity matters for two reasons. Bigger models do not fix this problem: scaling yields almost no gain on perspective-taking (Vision Language Models See What You Want but not What You See), and the best model on the controlled What'sUp benchmark reached 56% against 99% for humans (What's "up" with vision-language models?). Expensive verification also does not obviously help. In SpatialTraceGen, raising the verifier threshold improved trace quality scores by 17%, yet final accuracy on CLEVR-Humans stayed at 74% under every threshold. That result suggests the verifier rates how good the reasoning text looks, not whether the answer actually depends on correct geometry.
Core Claim
Spatial answers can be made both honest and cheap to supervise by placing a hard geometric bottleneck between perception and answering. A "hard" bottleneck means the answering module sees only the predicted geometry, never the pixels or the question text. Geometry is not automatically safe, though. When the bottleneck carries continuous coordinates, a jointly trained selector and answer head can hide the label in small coordinate offsets. Those offsets barely change box overlap with the true object, but they reliably signal "left" or "right." This is the spatial analogue of the position-index codes reported in Learning Robust Rationales for Model Explainability.
We predict that such codes concentrate in the bottleneck fields that no external tool anchors, such as facing direction, and in sub-pixel coordinate detail. We also predict that two existing anti-degeneration ideas transfer to this setting. The first is a frozen expert that never co-trains with the selector, following the alignment idea of Enhancing the Rationale-Input Alignment for Self-explaining Rationalization. The second is slowing the answer head's learning rate, following Decoupled Rationalization with Asymmetric Learning Rates. Together they should close these channels without sacrificing accuracy.
What is new relative to prior spatial work is the explicit bottleneck, the measurement of collusion through it, and its reuse as a data filter. SpaRE fine-tunes Qwen2-VL on 3.4 million synthetic spatial question-answer pairs with no bottleneck, so it cannot tell geometry from new caption shortcuts. SpatialTraceGen distills tool-assisted traces but verifies them with repeated language-model calls. Neither asks whether the student's intermediate evidence is causally used.
What is new relative to the rationalization literature is the setting itself. A2I, DAR, and DR all study text, and occasionally graphs. The DAR and DR papers both list vision and multimodal extension as open.
Research Program
The student. The student is Qwen2-VL-2B, chosen to match SpaRE's compact setting. It reads the image and question and, from its hidden states, regresses a structured rationale through small heads. For each object mentioned in the question, the rationale holds a box, a coarse depth bin, and one of eight facing-direction bins. A separate small answer network receives only these numbers and outputs the relation. Because the rationale is produced by regression heads rather than generated text, the whole system trains end to end with gradients.
Teacher signals from cheap tools. Box and depth targets are distilled from an off-the-shelf open-vocabulary detector and a monocular depth estimator. This is the same tool family SpatialTraceGen orchestrates, but run once per image rather than inside a multi-turn agent loop. Facing direction has no tool supervision. We leave it unsupervised on purpose, because it is the field where we expect hidden codes to form.
The frozen geometric expert. This is a small network trained only on procedurally generated layouts, where ground-truth relations are known exactly. It is then frozen. During student training, its answer on the student's rationale is added as an auxiliary agreement loss. Because the expert never adapts, a code the selector invents carries no meaning for it. This is DAR's mechanism moved from full-text experts to geometry.
Two further defenses. We apply DR's asymmetric learning rate, slowing the answer head relative to the selector. We also quantize the coordinates to a coarse grid, which caps how much information can hide in offsets.
The data filter. The same detector-plus-expert machinery filters SpaRE's synthetic question-answer pairs. A pair is kept when the expert, applied to detector geometry, agrees with the synthetic label. Pairs whose relation the expert cannot judge, such as orientation-dependent "facing away," are kept at reduced weight. The per-sample cost is one detector pass, one depth pass, and one small-network forward pass. We compare that cost against a language-model verifier.
Relation to distributed spatial signals. The Dual Mechanisms of Spatial Reasoning found that vision encoders spread spatial information across background regions. That finding seems to conflict with a bottleneck that discards background, but our design resolves it. The selector still reads the whole image and can use those distributed signals to place boxes. Only the answer must be expressible in object geometry. If some relations truly need background context the rationale cannot carry, that will show up as a per-relation accuracy gap between bottlenecked and unbottlenecked students. We will report that gap rather than hide it.
Baselines
Naive reference points. These are the untuned Qwen2-VL-2B and a Qwen2-VL-2B fine-tuned on an equal-size random subset of SpaRE pairs with no filtering. They establish what the data alone buys. The file's anchor numbers here are the published What'sUp best of 56% (BLIP fine-tuned on VQAv2) and zero-shot CLIP's 56% on VSR, both from the What'sUp and Visual Spatial Reasoning papers.
Strongest existing method. This is SpaRE-style fine-tuning: Qwen2-VL-2B trained on SpaRE data with SpaRE's own deduplication, consistency checks, and CLIPScore filtering. SpaRE reports a 49% improvement on What'sUp for the 2B model, an average 9% gain across VSR and 3DSRBench, and up to 4.9% on RealWorldQA. It is the strongest competitor because it uses the same model size, the same data source, and no bottleneck. We will run it ourselves on the identical retained data volume, with the same number of training steps and the same hyperparameter-search budget. Beating it means higher accuracy on What'sUp and on VSR's Concept Zero-Shot split at equal data. Alternatively, it means matching accuracy while showing a substantially lower collusion rate (defined below) and better counterfactual consistency. Matching SpaRE on in-distribution data alone would not count.
Verification baseline. This is a SpatialTraceGen-style filter in which an open language model grades each candidate sample. It is run on the same candidate pool, with cost measured in GPU-hours per retained sample. Beating it means equal or better downstream accuracy at a fraction of the cost. For context, SpatialTraceGen reports a 74% accuracy plateau on CLEVR-Humans, measured on only 100 images.
Soft-bottleneck ablation. This isolates the key ingredient. The student emits the same rationale, but the answer head also sees the full hidden state, which is essentially grounded chain-of-thought. The comparison tells us whether forcing answers through geometry, rather than merely producing geometry, is what matters.
Defense ablations. We train the hard-bottleneck student with no defenses, then add the frozen expert, the asymmetric learning rate, and quantization one at a time. This attributes any reduction in collusion to a specific mechanism.
Post-hoc steering reference. On COCO-spatial we include the encoder-amplification intervention of Cui et al., which reports corrections of up to 55% of errors and gains of up to 22 percentage points.
Why not just…?
Why not just fine-tune on SpaRE, which already improves What'sUp by 49%? This is the real threat. If plain SpaRE fine-tuning matches us on What'sUp, on VSR Concept Zero-Shot, and on counterfactual consistency, the bottleneck adds nothing for accuracy. Our remaining contribution would then be the cheaper filter. Our reason to expect a difference comes from Kamath et al., who found that spatial fine-tuning can lower controlled What'sUp performance through new shortcuts. SpaRE cannot test whether its gains survive identity-held-constant counterfactuals. The paired-image consistency metric described below settles this directly.
Why not just run a detector and hand-written geometric rules, with no learned student? For left/right and above/below on clean photos, such a pipeline may well match us. We will report it per relation. It cannot read facing direction, handle occlusion-based "behind," or resolve reference-frame ambiguity, which are the long-tail relations SpaRE identifies as hard. Its errors also compound from the detector. If it matches the student on all relations, we will conclude that the student's value lies only in the long tail.
Why not just have the model write boxes in its chain of thought, or reward them with reinforcement learning as in SpatialThinker? Boxes that the answer token can ignore may be decorative, because the answer still sees the full image and question. That leaves world-prior shortcuts such as the "on-table" bias in play. The soft-bottleneck ablation is exactly this alternative. If it matches the hard bottleneck on both accuracy and consistency, our central claim fails.
Why not just use spectral normalization instead of asymmetric rates? DR reports that rigid Lipschitz constraints, which cap how sharply a network's output can change with its input, cost accuracy, while asymmetric rates gained over 10% rationale F1 in text. We include spectral normalization on the answer head as a defense arm so the comparison is made in the spatial setting rather than assumed.
Proposed Experiments
Training data are SpaRE pairs drawn from DOCCI, Localized Narratives, and PixMo-Cap, together with procedurally generated layouts used only for the frozen expert. Evaluation covers What'sUp, VSR (both Random and Concept Zero-Shot splits), COCO-spatial and GQA-spatial, 3DSRBench, RealWorldQA, and CLEVR-Humans. MMBench serves as a forgetting check.
Collusion audit. This is the first decisive check. On What'sUp pairs, where the same objects appear in different relations, a grounded model must move its rationale when the answer changes. We define a collusion rate as the fraction of correct answers on which either of two things happens. The rationale geometry contradicts the answer under the frozen expert, or the rationale stays within quantization noise across a pair whose answer flips. We also perturb rationale coordinates by sub-box-size amounts and measure answer flips, which directly probes offset codes. The prediction is that the undefended hard-bottleneck student shows a high collusion rate, concentrated in the unsupervised orientation field and in fine coordinate detail. The prediction further holds that the frozen expert and coarse quantization cut that rate sharply while holding What'sUp accuracy. If no collusion appears even without defenses, the rationalization lens is unnecessary here, and we would report that.
Accuracy and transfer. This is the second decisive check. It compares the defended hard-bottleneck student against SpaRE-style fine-tuning and the soft bottleneck. We look in particular at the gap between VSR's Random split and its Concept Zero-Shot split, since a smaller gap indicates relations decoupled from object identity. We report per-relation results to locate where the bottleneck loses background context, as Cui et al. would predict.
Cost and quality of the filter. This is the third decisive check. At matched retained-set size, we compare students trained on geometrically filtered data, CLIPScore-filtered data, language-model-verified data, and unfiltered data. We report both accuracy and GPU-hours per retained sample. We also test whether the geometric filter's agreement score predicts downstream accuracy, whereas SpatialTraceGen's verifier score did not. If it does, the bottleneck has shown that an internal geometric consistency check, rather than an external judge, is what actually matters for spatial distillation.
References
- Vision Language Models See What You Want but not What You See
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- SpatialTraceGen: High-Fidelity Traces for Efficient VLM Spatial Reasoning Distillation
- Learning Robust Rationales for Model Explainability: A Guidance-Based Approach
- Enhancing the Rationale-Input Alignment for Self-explaining Rationalization
- Decoupled Rationalization with Asymmetric Learning Rates: A Flexible Lipschitz Restraint
- SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
- Adversarial Cooperative Rationalization: The Risk of Spurious Correlations in Even Clean Datasets
- The Dual Mechanisms of Spatial Reasoning in Vision–Language Models
- Visual Spatial Reasoning