Occlusion as a Retrieval Hop: Training 3D Agents on Questions Whose Evidence Is Hidden

Many questions about a room cannot be answered from where you stand. "What color is the mug on the shelf behind the open door?" has two steps. First you find the door. Then you move so the shelf is no longer blocked, and only then can you read the mug. In text question answering, a question that needs one fact to locate the next is called multi-hop. The standard fix there is to interleave reasoning with retrieval, so that the first step tells the system what to look up next. In 3D scenes, the thing you look up next is often a viewpoint, because the intermediate fact is physically hidden behind something closer to the camera. No existing benchmark or training set isolates this case. Existing 3D question-answering data also appears to be heavily solvable from room priors alone: on SQA3D, a blind text-only model scores 43.65% against 47.20% for a 3D model (SQA3D).

We propose building verified training and test data in which the intermediate hop of a question is occluded from the agent's starting pose. We do this by inserting counterfactual objects into real indoor scans at positions whose visibility is checked by construction. Human reviewers repair exactly the items where automatic depth ordering is unreliable or where the question leaks a shortcut. We then train a small vision-language model, using reinforcement learning with outcome-only reward, to treat "move the camera" as a retrieval action. The bet is that "look behind the occluder" can be learned as a reusable skill from synthetic data. Learning from Synthetic Data Improves Multi-hop Reasoning showed the analogous result for text, where chaining skill learned on fictional fact graphs transferred to real multi-hop benchmarks.

Core Claim

Occlusion-gated multi-hop questions are a distinct failure mode of current 3D reasoners. They can be synthesized at scale with ground-truth answers by inserting objects into real scans and checking visibility programmatically. Training on them should improve active 3D question answering on real benchmarks beyond what training-free active viewing achieves. Human effort helps only when it is spent on two specific failure classes, not spread uniformly over the data: depth-misregistered inserts, and questions a blind model can already answer. All three parts of the claim are falsifiable, and the experiments below are designed so that each one can fail independently.

What Exists and Where This Differs

Three papers supply the motivation.

CoV: Chain-of-View Prompting for Spatial Reasoning runs a think-act-observe loop in which a frozen vision-language model picks anchor views and then nudges the camera to uncover occluded context. It gains an average of +11.56% LLM-Match across four models on OpenEQA and reaches 51.1 EM@1 on SQA3D. CoV shows that active viewing matters. It is training-free, however, so its camera moves are only as good as the frozen model's guesses. We train the viewing policy on questions where the move is provably necessary.

SQA3D and ScanQA supply situated and object-grounded 3D questions over ScanNet. Both were built by crowdsourcing, both are static, and neither controls whether intermediate entities are visible.

SIG3D improves situation grounding with voxel anchors, reaching 52.6 EM@1 on SQA3D. It still answers from a fixed scan and never needs to move.

Two further papers supply methods.

Learning from Synthetic Data Improves Multi-hop Reasoning contributes the training recipe: GRPO with outcome-only reward on rule-generated questions, reporting 56% to 131% relative F1 gains on real multi-hop benchmarks. The same paper also reports a ceiling, because synthetic templates underperform exposure to real linguistic variety. That ceiling is why we do not rely on templates alone.

V2XScene contributes the idea of inserting assets into real imagery with programmatic occlusion and collision checks. It uses this only to train a driving detector, not to generate questions or reasoning chains.

Two papers constrain the design.

Virtual Occlusions Through Implicit Depth is the source of the depth-ambiguity lens. That paper shows that per-pixel depth comparison against reconstructed geometry produces bleeding, flickering occlusion edges, especially at thin structures and transparent or specular surfaces. Its implicit binary occlusion query cut flicker by nearly 30% on ScanNetv2. Inserting a virtual object into a real scan is exactly an augmented-reality compositing problem, so our synthesized scenes inherit these errors. An inserted mug that bleeds through a chair leg quietly leaks the hidden answer.

Disentangling Extraction and Reasoning in Multi-hop Spatial Reasoning warns that synthetic spatial data can be misleading. Its symbolic reasoner scores 96 to 97% on programmatic SPARTQA-Auto but lags neural models on real RESQ, where its best neural model reaches 69.5%. We therefore evaluate transfer only on real, human-written questions.

Research Program

Scene synthesis. We start from ScanNet scans with their reconstructed meshes and camera trajectories. For each scan we choose an occluder that already exists in the scene, such as a door, cabinet, sofa back, or monitor. We then place a new object from a small asset library at a physically valid location behind it: resting on a surface and not intersecting other geometry, using separating-axis collision checks in the spirit of V2XScene. We then sample an agent start pose and a situation description in SQA3D style. Rendering is checked at three levels:

  1. The inserted object must be under a small visible-pixel fraction from the start pose.
  2. It must be clearly visible from at least one reachable pose.
  3. The occluder must be visible from the start pose.

Questions are generated from a grammar over scene relations. The first hop names the occluder by relation to the agent, the second hop asks about the hidden object, and some questions add a third hop to an object near it. Because the hidden object's identity, color, and count are drawn at random, room priors cannot predict the answer. This counterfactual control is what inserting objects buys us over mining naturally occluded ScanNet objects. Answers are correct by construction.

Where humans enter. Automatic generation fails in two predictable ways, and humans review only those two queues.

The first queue is depth misregistration. ScanNet meshes have holes and poor geometry on glass and thin parts, so an insert may bleed through its occluder or float. We flag items where mesh-depth compositing and an implicit-depth occlusion mask in the style of Watson et al. disagree near the insert's boundary. Reviewers accept, adjust, or reject these items.

The second queue is shortcut leakage. We run a blind text-only model and a single-start-view model on every item, and send any item either model answers correctly to reviewers. Reviewers rewrite the question to remove the leak and paraphrase it into natural language.

This follows the author-reviewer pattern, in which reviewing model drafts was about 13 times more efficient than authoring (Building a Role Specified Open-Domain Dialogue System). It also follows VisIT-Bench's finding that targeted, instruction-specific human descriptions beat generic ones by a wide margin, 91% versus 31% evaluation success (VisIT-Bench). We fix the human budget in hours in advance and log it.

Held-out real evaluation. Separately, annotators write occlusion-gated questions directly on held-out ScanNet scenes with no insertion. Each question must refer to real objects that are not visible from a specified start pose. This set never touches the synthesis pipeline and is the main test of transfer.

Agent and training. An open vision-language model of a few billion parameters acts in the same rendered-view environment CoV uses. It can emit reasoning text, camera translations and rotations, or a final answer. We train it with GRPO. The reward is answer correctness minus a small per-move cost, and there is no step-level supervision.

Baselines

The naive reference points are the same base model answering blind, from the situation text only, and answering from the single start view. The blind reference also defines the ceiling of prior-only guessing on each test set; on SQA3D the reported blind score is 43.65% (SQA3D).

The second baseline is static dense multi-view input: the base model receives a fixed budget of views sampled from the scan trajectory. It tests whether "just show more frames" suffices. We match its view count to the average number of views our agent actually uses.

The strongest active competitor is CoV run on our exact base model, environment, and view budget. It is the most direct threat because it already moves the camera to reveal occluded content, and its reported numbers are 51.1 EM@1 on SQA3D, 116 CIDEr and 31.9 EM@1 on ScanQA, and a +11.56% average LLM-Match gain on OpenEQA (CoV). Its absolute OpenEQA score is not given in our file, so we rerun it ourselves. Beating CoV means a statistically clear gain on our held-out real occlusion set and a non-negative change on SQA3D and OpenEQA with the same base model and budget.

The strongest reported static model on SQA3D is SIG3D at 52.6 EM@1 (SIG3D). We report against that published number rather than retraining it. We would count the result as competitive if our agent reaches or exceeds it while using a much smaller base model and no 3D-specific encoder.

To make the comparison fair on data, a fine-tuning baseline trains the same model with the same GRPO recipe and step budget on SQA3D's own training questions instead of our synthetic data.

Two ablations isolate the key ingredients.

The first is a visible-hop control: the identical synthetic pipeline and question count, but with the inserted object placed where it is already visible from the start pose. If this matches the full method, the occlusion structure is not what matters.

The second is a data-source ladder at a matched item count:

  1. programmatic-only;
  2. programmatic plus LLM paraphrase;
  3. programmatic plus LLM paraphrase plus targeted human review;
  4. naturally occluded ScanNet objects without insertion.

This ladder separates the value of counterfactual insertion from the value of human review.

Why not just…?

Why not just use CoV with a stronger frontier model? This is the most serious threat. CoV's gains come without any training, and scaling its test-time move budget adds a further +2.51% on average. Our position is that a frozen model's camera moves are heuristic. Its reported ablation, a ~4.6% drop when initial view selection is skipped, shows that performance depends heavily on the choice of views. We cannot cite evidence that frozen models fail specifically on occluded intermediate hops. The held-out real occlusion set, with CoV on the same model and budget, answers the question directly. If CoV matches our trained agent there, the contribution shrinks to the benchmark and the shortcut audit, and we will report it that way.

Why not just caption the scene into text with object coordinates and let an LLM reason? SQA3D's text-based GPT-3 baseline reached 41.0%, below even the 43.65% blind baseline. The SQA3D authors attribute this to captions discarding geometric detail. Captions also presuppose that a detector has already seen the hidden object, which is exactly the condition that fails in our setting. We still include a caption-plus-coordinates baseline built from the full-scan object list. That version cheats by seeing everything, so it serves as an upper bound on what perception-free reasoning could extract.

Why not just generate the data fully automatically, since rule-based synthesis already works for text? The synthetic multi-hop paper itself reports a gap relative to training on real data, and SPARTQA-Auto versus RESQ shows how badly programmatic spatial data can mislead. In 3D there is an extra failure mode that paraphrasing cannot fix: compositing errors from depth misregistration can leak the hidden answer into the start view. Whether humans are needed at all is an empirical question. If rung 2 of the data-source ladder (LLM paraphrase) equals rung 3 (human review) on the held-out real set, we drop humans from the paraphrase role. In that case we keep humans only for the depth-misregistration queue, and only if that queue measurably reduces leakage.

Why not just mine naturally occluded objects in ScanNet instead of inserting new ones? This option is cheaper and avoids compositing error entirely. The risk is that naturally hidden objects are predictable from room type, which is exactly why blind baselines score well. Rung 4 of the ladder, evaluated on the held-out set, tests this directly.

Decisive Checks

We will first confirm that the held-out real occlusion set is actually hard for the naive baselines. Blind and single-view accuracy should sit near chance, while a model given an oracle revealing view should do well. If they do not, the set does not isolate occlusion and must be rebuilt.

The central result is the trained agent against CoV on that set. We will also measure two diagnostics:

  1. Hop-2 accuracy conditioned on whether the agent reached a pose where the hidden object is more than half visible. This separates failures of looking from failures of reasoning.
  2. The number of camera moves per question.

Transfer is measured by EM@1 on SQA3D, EM@1 and CIDEr on ScanQA, and LLM-Match on OpenEQA. Each is compared against CoV reproduced on the same model and against the published SIG3D and CoV numbers.

The visible-hop ablation decides whether occlusion structure is the active ingredient. The data-source ladder decides whether targeted human review earns its cost; we report accuracy per human-hour. Finally, we report the leakage rate: the fraction of synthesized items a blind model solves before and after the depth-misregistration and shortcut review queues. This measures directly whether the depth-ambiguity problem from augmented reality compositing contaminates synthetic 3D reasoning data.