Teaching Spatial Models What to Keep: Evidence-Labeled Synthetic Scenes for Budgeted Multi-View Reasoning

A question like "standing at the fridge, is the chair to my left?" about a room often depends on a few image regions spread across dozens of camera views. Vision-language models, which read images as long sequences of patch tokens, cannot fit every view into their input. Models that are handed many views tend to ignore most of them anyway. Current fixes all run at test time on a frozen model. One renders new views on demand from a simulator. Another drops patches whose 3D location has already been seen. Neither teaches the model to decide which visual evidence a spatial question actually needs.

We propose to use rendered 3D scenes for a property that web data cannot provide. When we generate a spatial question synthetically, we also know exactly which pixels in which views contain the evidence that answers it. We will generate multi-view spatial question-answer data in which every question carries a per-patch evidence label. We will then train a model to choose a fixed budget of visual tokens from a pre-recorded pool of views and to answer from only that budget. The claim is that question-conditioned evidence supervision, which only synthetic generation can supply at scale, produces models that degrade much more slowly as the token budget shrinks than models that prune by geometric redundancy or by chance. It also requires no simulator at test time.

Core Claim

Three bodies of work frame the gap. First, synthetic spatial data helps but stops at single images or single objects. SpaRE mines dense captions into 3.4M spatial question-answer pairs and fixes the long tail of spatial relations, and Ultimate3D renders Blender priors through a ControlNet to teach camera-object relations. Both explicitly leave multi-view scenes open. CVVG-Reasoner builds multi-view spatial data from video using segmentation and depth models, but it needs heavy manual cleaning and never limits the visual tokens it consumes.

Second, context-budget methods for 3D question answering are training-free. Chain-of-View prompting selects anchor views and then moves a virtual camera inside a 3D scene representation. Online geometry-aware token pruning removes patches whose voxels were already observed. Its authors concede that the removed patches may carry the subtle perspective cues a question needs.

Third, the attention-gap analysis shows that image tokens fill about 90% of the input yet receive about 10% of attention. This motivates fewer, better-chosen visual tokens rather than more.

Our proposal differs from each in one specific way. Unlike SpaRE and Ultimate3D, the data is multi-view and labeled with evidence locations. Unlike CVVG-Reasoner, its labels come from exact scene geometry rather than noisy depth lifting. Unlike CoV and geometry-aware pruning, the selection is learned, conditioned on the question, and runs on static recordings. The pruning criterion is "relevant to this question," not "not yet seen."

Research Program

Data generation. The generator starts from 3D scenes whose geometry and object instances are known exactly. These come from HM3D meshes and from procedurally arranged Blender rooms in the style of Ultimate3D's rendering stage. For each scene we render a camera trajectory of 30 to 60 views with exact poses, depth, and instance masks.

Questions are produced programmatically from the scene's object graph. They cover relative direction, distance, counting across views, orientation ("is the sofa facing the window?"), and situated questions in the SQA3D style, where the asker's position and heading are given. Frame-of-reference variants vary whether "left" means viewer-centric or object-centric, which is the ambiguity SpaRE lists as unresolved. An LLM only paraphrases templates; it never decides answers, so answers are correct by construction.

Evidence labels. For every question, the objects and the situational anchor it depends on are projected into every view. Each patch is labeled as evidence if it covers one of them. We deliberately oversample "redundancy traps." These are questions whose deciding cue, such as an object's front face, is visible only in a later view of a region already seen from behind. On these questions, overlap-based pruning should fail by design.

An ablation compares raw renders against ControlNet-restyled renders. It tests whether photorealism matters for transfer to real scans, given the known risk of rendering artifacts.

Model. We start from Qwen2.5-VL-7B, the same base model used in the geometry-aware pruning paper, so its reported numbers are a direct reference. The vision encoder processes all candidate views, which is cheap relative to the language model. A small scoring head conditioned on the question text scores every patch. The top $B$ patches, with their view index and position encodings, enter the language model.

Training minimizes the answer loss plus a binary cross-entropy loss between patch scores and evidence labels:

$$ \mathcal{L} = \mathcal{L}_{\text{answer}} + \lambda \, \mathcal{L}_{\text{evidence}} $$

Here $\lambda$ weights the evidence term. The budget $B$ is sampled per batch between 256 and 4,096 tokens, so a single model serves any budget at test time. Gradients reach the scorer through the evidence loss directly, and through the answer loss via a straight-through top-$B$ estimator.

Mechanism analysis. Beyond accuracy, we measure the share of attention the language model places on visual tokens, using the measurement protocol of the attention-gap paper. We also measure evidence recall: the fraction of ground-truth evidence patches that survive selection. Together these test whether evidence-supervised selection closes the 90/10 attention gap or merely shortens the input.

Baselines

Blind and naive references. The do-nothing reference is the base Qwen2.5-VL-7B fed uniformly subsampled frames at each budget, together with the published blind scores. SQA3D's blind text model reports 43.65% and its best model 47.20% (SQA3D). OpenEQA's blind GPT-4 scores 33.5% against roughly 50% for GPT-4V (OpenEQA). Any method that sits within a few points of the blind floor is not using vision.

Geometry-aware token pruning. This is the strongest competitor that operates under the same constraints: static recordings, a token budget, and the same base model. Online geometry-aware token pruning reports that Qwen2.5-VL-7B goes from 24.1 to 25.1 EM on ScanQA and from 46.5 to 47.3 EM on SQA3D, while reducing tokens. We run it in two forms.

The first form applies it to the base model, which reproduces the paper's setting. The second applies it to a model fine-tuned on our same synthetic data without evidence supervision. That second form is the fair version: identical data, identical training steps and tuning budget, with the pruning threshold $\tau_0$ tuned to hit each budget $B$.

Beating it means higher exact match at every matched budget on ScanQA and SQA3D. The gap must be largest at small budgets and on the redundancy-trap subset.

Chain-of-View. CoV reports 116 CIDEr and 31.9 EM@1 on ScanQA, 51.1 EM@1 on SQA3D, and an average gain of +11.56% LLM-Match on OpenEQA. It is the best reported system here, but it renders new views from a 3D representation at test time, which our setting forbids.

We will report its numbers as an upper reference, not as a matched comparison. Separately, we will run its anchor-view-selection stage alone, without active camera moves, on the same static pool, at matched tokens. Matching CoV's full numbers without a simulator would be a strong result. Exceeding the selection-only variant is the fair bar.

Ablations isolating the key ingredient. Three variants share the same data and compute. The first trains with random token dropout at the same budgets. The second trains with overlap-based dropout following Lai et al.'s criterion. The third trains with the answer loss only, with no evidence loss. If the evidence loss is not responsible for the gains, at least one of these will match the full model.

Why not just…?

Why not just extend the context window and feed every view? Position-encoding extensions like LongRoPE make long inputs fit, but fitting is not using. The pruning paper already finds that removing redundant tokens slightly improves ScanQA exact match. The attention-gap analysis shows the extra image tokens are largely ignored. Results from code and relational models (RepoDebug, Context Window Failures in Relational Foundation Models) show the same degradation or out-of-memory failures as inputs grow. We include the full-context base model as the right end of every budget curve. If it beats our model at matched compute, the claim fails.

Why not just fine-tune on multi-view spatial data and prune at test time? This is the most serious alternative. It is exactly the second form of the geometry-aware pruning baseline above, run on our own data, so the experiment settles it directly. Our prediction is specific. The two should be close when the budget is generous. They should diverge on redundancy-trap questions and at small budgets, because overlap pruning discards the cues those questions hinge on. If they do not diverge, the evidence labels are not worth their cost and we will say so.

Why not just convert the scene to a text map of 3D coordinates? Ego3D-VLM shows that text coordinates beat rendered bird's-eye-view maps and are cheap in tokens. It is a real threat on metric distance questions. However, it inherits every miss of its detector and depth estimator. Text summaries also lose orientation and situational nuance, which is the failure SQA3D documents for language-only models.

We will run a text-map baseline built from the same external tools on SQA3D and OpenEQA at matched token counts. We will also run a hybrid in which our selected tokens are concatenated with the text map. If the hybrid wins, the right conclusion is that learned selection and symbolic maps complement each other.

Why not just reweight attention at test time? ADAPTVIS raises LLaVA-1.6 from 48.2% to 98.2% on a controlled WhatsUp subset. However, it was evaluated on two-object single images, and it cannot recover evidence that is absent from the input. Under a token budget, the issue is which tokens are present at all. We will apply it on top of both the pruning baseline and our model to check whether the two effects add.

Proposed Experiments

Main evaluation. The primary result is accuracy as a function of token budget. We report ScanQA exact match and CIDEr, SQA3D exact match, and OpenEQA LLM-Match on its episodic-memory split, which has a reported Spearman correlation of 0.909 with human judgment. Each is measured at budgets from 256 to 4,096 visual tokens. We summarize the curves by their area and by the accuracy retained at the smallest budget.

Leakage control. HM3D scenes appear in OpenEQA, so we hold out every OpenEQA-HM3D scene from generation and also report results on the ScanNet portion separately. ScanQA and SQA3D are built on ScanNet, which our generator never uses, so performance there measures transfer from synthetic to real scans.

Out-of-domain transfer. We probe transfer on Ego3D-Bench, a multi-camera driving benchmark, and on MV3DSR-Bench. Results on these are not expected to be strong and are reported as such.

Regression check. We verify that single-image spatial skill and general skills do not regress on WhatsUp, VSR, and POPE. SpaRE's reported 49% WhatsUp gain gives the reference for what single-image spatial fine-tuning can achieve.

Decisive checks. Three results decide the claim. First, the full model beats the answer-loss-only and overlap-dropout ablations on the redundancy-trap subset of a held-out synthetic test set, and at budgets of 1,024 tokens or fewer on SQA3D. Second, evidence recall on synthetic data predicts accuracy on real benchmarks across models; if it does not, the evidence labels are not tracking what matters. Third, the visual share of attention rises well above the roughly 10% baseline under our model, but not under random-dropout training at the same budget. If the first check fails, the proposal reduces to "more multi-view synthetic data helps," which CVVG-Reasoner already shows.