Repairing Spatial Reasoning from the Bottom Tier Up

Vision-language models, meaning models that read images and answer questions about them in text, often describe a scene fluently and then get "is the cup left of the laptop from the woman's point of view?" wrong. Fine-tuning helps on the benchmark being tuned for. Yet when the same problems are re-rendered from a new camera angle or with new parameters, accuracy falls sharply. The field cannot currently say where the failure sits. The model might misperceive the objects, misjudge the relation between them, or botch a multi-step inference built on correct relations. It also cannot tell whether the benchmark gains came from skill or from memorized appearance.

This note proposes a training loop built around a single diagnostic. A procedural 3D scene renderer produces questions at three tiers: perceive the objects, relate them in a stated frame of reference, and compose several relations into an answer. The renderer knows the exact answer at every tier. That lets us feed the model the correct lower-tier answers and see whether it then succeeds at the tier above. Each round, the loop finds the lowest tier that is still broken for each kind of question and generates the next round's training data there. The claim is that aiming new synthetic data at the lowest failing tier beats generating the same number of examples uniformly. The benefit should show most clearly on held-out camera viewpoints, object categories and observer frames that were sealed off from training, where memorization cannot help.

Core Claim

Two findings in the file motivate the proposal and appear to conflict. Beyond Memorization rewrote text benchmark items as synthesized multi-step reasoning questions. This removed post-cutoff performance decay and cut the rate at which influence functions trace a question to its source document from 77.5% to 17.5%. So multi-step reasoning looks like a defense against contamination. Yet DynaSolidGeo re-renders multi-step solid-geometry problems with new parameters and viewpoints and still finds a 20.4% drop for Claude-Sonnet-4.5, with visual perception errors behind more than half of Gemini-2.5-Pro's failures.

Our reading is that memorization in vision-language spatial reasoning lives mainly at the perceptual tier, below where linguistic rewriting can reach. If that is right, two predictions follow. First, a model's drop from familiar to perturbed inputs should mostly vanish once correct perception is supplied to it. Second, training data aimed at the perceptual and relational failures that this intervention exposes should repair the drop more efficiently than uniform data. Both predictions can fail, and the experiments below are built to test them.

What Exists and What Is Missing

The closest prior system is SpatialThinker. It trains a 7B model with group-relative policy optimization, a reinforcement learning method that scores each sampled answer against the others drawn for the same prompt. Its reasoning scratchpad runs through observe, scene-graph, think and answer stages, and it gates rewards so that format and object counting must be right before localization is rewarded. It reports a 12.1% gain over GPT-4o on 3DSRBench. We reuse its reward design unchanged. What we change is its training set, the static STVQA-7K, and we evaluate on sealed generator regions instead of only fixed benchmarks.

SpaRE mines 3.4 million spatial question-answer pairs from dense captions in a single pass and reports a 49% gain on What'sUp. Its labels come from language rather than geometry. Its questions are flat, with no tiers, and it never revisits the data after seeing where the model fails. We use it as the representative of large single-pass data.

Three works supply methods. A²RBench shows the value of generation that verifies its own outputs, and it also shows the limit. Its author model simplified its logic when asked to produce 3D rules, which produced a V-shaped difficulty curve. We avoid that ceiling by having a deterministic renderer compute every answer, so no language model is asked to reason about 3D in order to write the labels. DyVal shows, in text-only graph reasoning, that dynamically generated problems expose models that look strong on static tests, and that training on generated problems transfers. We carry that logic into images and add iteration. Spend Wisely proves that a constant number of synthetic samples per round stalls at a gradient-noise floor, while geometric growth converges. It reports 20.65% versus about 18.8% on GSM_p2. We adopt its growth schedule as a default and test it once in the spatial setting.

VL-Cogito orders training data from easy to hard. That is a difficulty curriculum, not a diagnosis of which competence is broken, and we test the difference directly.

The motivation also rests on several diagnostic studies. What's "up" finds spatial prepositions in fewer than 0.2% of web captions, and finds that fine-tuning on static spatial data can reduce accuracy on its controlled image pairs. Visual Spatial Reasoning shows that models drop on object concepts unseen in training. Vision Language Models See What You Want but not What You See reports that no model exceeds chance, about 29%, on perspective-taking. Together these say that static fine-tuning teaches shortcuts, and that frames of reference (camera, object, other observer) are the hardest unsolved case.

Research Program

The generator samples indoor 3D scenes, starting from HSSD-style synthetic assets like those behind 3DSRBench-synthetic. It varies object sets, camera elevation and azimuth, occlusion, and the presence of a designated observer. From the scene's true geometry it writes questions at three tiers:

  • Tier one asks about presence, counts, 2D boxes and depth order.
  • Tier two asks about pairwise relations in an explicitly stated frame of reference: camera, object-intrinsic, or another observer's.
  • Tier three asks multi-step questions such as chained relations, perspective-taking, fit and collision.

Every tier-three question is built from specific tier-two facts, and each tier-two fact from tier-one facts. Because of that dependency structure, the lower-tier ground truth can be written into the prompt. That makes the "oracle intervention" possible: we measure tier-$k$ accuracy both with the model's own lower-tier outputs and with the true ones. The difference tells us whether the failure originates at that tier or cascades up from below. This addresses the cascade problem that makes existing hierarchical benchmarks hard to interpret.

Probe results are grouped into cells. A cell is a tier combined with a relation type, a frame of reference and a viewpoint bin. For each cell $c$ in round $t$, let $a_c^{\text{oracle}}$ be accuracy when the lower-tier truth is supplied and $a_c^{\text{own}}$ be accuracy when the model relies on its own lower-tier answers. The failure attributed to the cell itself is $1 - a_c^{\text{oracle}}$; the gap between the two is cascaded error charged to the tier below. Round $t$ then generates

$$n_c^{t} = n_0 (1+u)^{t} \cdot \frac{\epsilon_c^{t}}{\sum_{c'} \epsilon_{c'}^{t}},$$

where $n_0$ is the first-round budget, $u$ is the growth rate taken from Spend Wisely, and $\epsilon_c^{t}$ is the cell's own attributed error. A small floor keeps every cell sampled, which guards against forgetting. Training in every round uses SpatialThinker's gated rewards unchanged, so the data selection rule is the only thing that differs.

For contamination control, we seal whole regions of the generator's parameter space before training begins: an elevation band, a set of object categories, and the observer-frame combinations. These regions are never sampled during training. Sealed-region accuracy therefore measures transfer, not recall, and the gap between trained and sealed regions serves as a direct "memorization tax."

Baselines

The naive reference is the base model, Qwen2.5-VL-7B-Instruct, the same base VL-Cogito uses, with no spatial training. It sets the floor on every benchmark and supplies the first round's probe diagnosis.

The strongest existing method is SpatialThinker, retrained from the same base on STVQA-7K with its published reward gating. The file reports a 12.1% gain over GPT-4o on 3DSRBench, 76.3% on RoboSpatial-Home and 65.9% on MM-Star. No absolute 3DSRBench accuracy for SpatialThinker was found on Lacuna, so we will rerun it ourselves rather than quote a number. It is the strongest competitor because it already uses a tiered perception-to-reasoning structure and gets large gains from little data. It is exactly the method that could make iteration look unnecessary. Beating it means higher accuracy on sealed regions, PerspectBench and 3DSRBench's uncommon-viewpoint split at equal policy-optimization compute, with RoboSpatial-Home and MM-Star not falling by more than one point.

The large single-pass data baseline is SpaRE, trained on its released 455k-sample set with the same optimizer and step budget. Its reported result is a 49% gain for Qwen2-VL-2B on What'sUp and about 9% averaged over VSR and 3DSRBench.

The ablation that isolates our key ingredient is uniform single-pass renderer data. It uses the same generator, the same total sample count as our full loop summed over rounds, the same rewards and the same compute, drawn uniformly over cells in one pass. A second ablation keeps the iteration but allocates new samples by final-answer error only, with no oracle-intervention attribution. This tests whether the tier diagnosis matters or whether iteration alone does the work. A third variant replaces attribution with a VL-Cogito-style easy-to-hard curriculum over the same generator. A last run compares constant-per-round budgets against geometric growth.

All variants share the base model, the total rendered sample budget, the GPU-hours, and a tuning budget of equal size spent on a held-out slice of the unsealed generator region.

Why not just…?

Why not just render a very large uniform dataset once? This is the real threat. The renderer already guarantees correct labels, and most of any gain may come from that alone. The uniform single-pass ablation, at matched samples and compute, settles it. If it matches the targeted loop on sealed regions and PerspectBench, we will report that tier targeting adds nothing. The contribution then shrinks to the oracle-intervention diagnostic and the sealed-region evaluation protocol, which remain useful on their own.

Why not just train SpatialThinker on more data? More static data from the same distribution is essentially our uniform ablation with extra steps. What'sUp's finding that static spatial fine-tuning can hurt on controlled images, and VSR's concept zero-shot drop, both suggest that volume without targeting learns shortcuts. The sealed-region gap measures this directly.

Why not just use an easy-to-hard curriculum? VL-Cogito's curriculum moved a vanilla GRPO average from 58.1% to 59.5%, a modest gain. Difficulty order does not distinguish a model that misperceives from one that misrelates. The curriculum baseline is run on the same generator, so any advantage of attribution appears head to head.

Why not just scale the model? Gao et al. find perspective-taking flat or falling as model size grows, with no model above chance. 3DSRBench shows GPT-4o and Gemini-1.5-Pro losing 13.5% and 32.2% on uncommon viewpoints. We include GPT-4o probed on our sealed regions as a scale reference, not as a trained competitor.

Proposed Experiments

The first experiment is purely diagnostic and needs no training. We run the oracle-intervention probe on the base model, SpaRE and SpatialThinker, comparing trained-like versus sealed viewpoints. We also rerun DynaSolidGeo's static-versus-dynamic comparison, supplying vertex and edge identifications drawn from its rendering scripts wherever those scripts expose them. The prediction is that supplying perception removes most of the static-to-dynamic drop. If it does not, and the drop remains at the relation or composition tier, the bottom-up allocation rule is the wrong default. We would then invert it before training.

The main experiment trains all variants for five rounds under matched budgets. We report per-tier accuracy on sealed regions and the memorization tax, meaning trained-region accuracy minus sealed-region accuracy. We also report What'sUp, VSR on its concept zero-shot split, 3DSRBench overall and uncommon-view with FlipEval, PerspectBench against its roughly 29% chance level, RoboSpatial-Home, and DynaSolidGeo's process-qualified accuracy on dynamic instances. MM-Star is the check that general ability survives.

The decisive result would have three parts. The targeted loop beats both uniform single-pass data and final-answer-only iteration on sealed regions and PerspectBench. It shrinks the memorization tax more than it raises trained-region accuracy. And its gains come mainly from cells the diagnostic flagged at the lowest tiers. We will check that attribution by comparing per-cell gains against allocation weights across rounds.