Seeing From There: Distilling Hidden Geometry Into a Late Feedback Pass for Perspective Taking Behind Occluders

Vision-language models fail at a simple question: can that person see the cup? To answer it, a model has to do two things a single camera image does not support. It must infer geometry it cannot see, such as the part of the shelf hidden behind a box. It must also re-express that geometry from someone else's position. Benchmarks show both failures at once. On the 3D-PC, fine-tuned networks reach human level on basic line-of-sight judgments but collapse to 66% once targets move behind occluders, against 87% for humans, because they learned 2D size-and-position shortcuts (The 3D-PC). Frontier models fall from above 90% on recognition to 50–60% on perspective taking (Seeing Through Their Eyes). Current methods for injecting 3D knowledge into 2D models cannot fix this as designed. They distill depth or 3D features from a teacher that sees only the camera's visible surfaces, so the student learns a 2.5D world in which hidden space is empty.

We propose to train a single-image student against a privileged teacher that has seen the scene from other viewpoints, including the agent's own. The hidden-geometry knowledge it is taught enters through a late, query-conditioned feedback pass, not through the initial feedforward computation. The design follows the visual cortex. There, a fast feedforward sweep reports local visible features, and only afterwards does higher-level context write object completion and figure-ground information back onto early, high-resolution maps (Early Visual Cortex as a Multiscale Cognitive Blackboard). In two sentences: teach the model what is behind things using a teacher that actually saw behind them, and make that knowledge a second, viewpoint-specific pass that adjusts the visible features without replacing them. The test is whether this beats both simpler options, a stronger feedforward student and an explicit symbolic geometry pipeline, specifically on occluded perspective-taking queries.

Core Claim

The perspective-taking failures behind occluders do not come mainly from missing data or model scale. They come from what the 3D supervision contains. Spatial Forcing aligns intermediate tokens of a 2D vision-language-action model with features from the 3D geometry model VGGT, using a cosine loss (Spatial Forcing). The camera-aware spatial model distills point clouds from the monocular depth model UniDepth v2 into visual tokens (On the Generalization Capacities of MLLMs for Spatial Intelligence). Both teachers describe the scene as seen from the student's camera. Neither carries information about surfaces hidden from that camera, which are exactly the surfaces that decide whether another agent's line of sight is blocked.

We make two claims. First, a teacher that is fed additional views of the scene, so that its features encode the hidden geometry, transfers that information to a single-view student better than a camera-view teacher does. Second, the transferred information is learned and used better when it is written by a late modulatory pass conditioned on the queried viewpoint. Here "modulatory" means the feedback rescales existing features and cannot create activity from nothing, as cortical feedback onto superficial layers does.

These claims make predictions that could fail. The gain over a compute-matched feedforward student should be concentrated on occluded and agent-frame queries. Accuracy on visible, camera-frame queries should not drop.

The feedback update is written as

$$h_i^{(1)} = h_i^{(0)} \odot \big(1 + \tanh(g_\theta(h_i^{(0)}, c, q))\big),$$

where $h_i^{(0)}$ is the feedforward feature of image token $i$, $c$ is a pooled global context from the encoder's top layer, $q$ is an embedding of the queried viewpoint (camera, or an agent pose given by a marker in the image), $g_\theta$ is a small cross-attention network, and $\odot$ is elementwise product. The bounded multiplicative form keeps the visible-surface code intact and directly tests the "contrast problem" raised in the blackboard review: can strong bottom-up signals swamp top-down writes? Distillation is applied only to the post-feedback tokens:

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_i \big(1 - \cos(W h_i^{(1)}, t_i^{\text{mv}})\big),$$

where $t_i^{\text{mv}}$ is the teacher feature at token $i$'s location, computed by VGGT from the student's image plus held-out views of the same scene, $W$ is a learned projection, and $\lambda$ weights the alignment term.

Research Program

Training data comes from scenes that have both multiple views and metric 3D structure. ScanNet supplies meshes and many frames per room. Co3D supplies object-centric multi-view captures, which the 3D-PC already turns into Gaussian-splat environments. For each training example, the student sees one frame with an inserted agent marker at a sampled pose. The teacher sees that frame plus other frames of the same scene, chosen to include a frame near the agent's pose and frames that view the student camera's occluded regions.

Ground-truth labels come from ray casting against the mesh or splat. For each target object, we record whether the agent can see it, what fraction of it is visible to the agent, and its direction in the agent's frame. We call this held-out evaluation set Seen-From-There. It is built over test scenes never used for training, so it measures the gap the file identifies as unaddressed: amodal perception in an agent's frame rather than the camera's. Amodal perception here means reasoning about parts of objects that are hidden.

The student is Qwen2.5-VL-3B, the base model of MVSM (ViewSpatial-Bench). This makes comparison with the strongest fine-tuned perspective-taking result direct. The feedback module sits on the vision encoder's output tokens before they enter the language model. One feedback iteration is the default, and two iterations are tested as an ablation.

We pick the alignment layer and the strength $\lambda$ on a validation split. We expect the choice to matter, because Spatial Forcing found a single middle-to-late layer optimal and later layers disrupted action generation.

The neuroscience literature supplies the design and a diagnostic, not a claim of biological fidelity. The modulatory-feedback ventral-stream model shows how to train feedback for segmentation on top of fixed feedforward weights, though only on SegMNIST (Modulatory feedback determines attentional object segmentation). We mirror its staging: first fine-tune the feedforward student, then train the feedback pathway with the encoder mostly frozen. This keeps the two contributions separable. The top-down VAE shows that hierarchical priors produce delayed filling-in of illusory contours (Top-down perceptual inference). That motivates our analysis question: does occluded-region information become linearly decodable only after the feedback pass and not before it?

Baselines

The naive reference is the untouched Qwen2.5-VL-3B, prompted directly and with chain-of-thought. Its reported ViewSpatial-Bench accuracy is 35.85% (derived from MVSM's reported 82.09% and 46.24-point gain), and frontier models sit near chance on the 3D-PC basic line-of-sight task (50–53%).

The strongest fine-tuned competitor is MVSM, the same 3B model fine-tuned on 43,000 automatically generated multi-perspective samples with a frozen vision encoder. It reports 82.09% overall accuracy on ViewSpatial-Bench. We will retrain it on our training scenes with the same question templates, the same number of optimizer steps, and the same hyperparameter-search budget as our method. This makes it the "just do supervised fine-tuning" control on identical data. Beating it means matching it within one point on ViewSpatial-Bench while exceeding it on the occluded subsets described below.

On the occluder-specific task, the reference number is the 3D-PC's fine-tuned Swin Transformer at 66% on VPT-Strategy, against 87% for humans. On occlusion understanding, the reference is Gemini-2.5-Pro's 62.2% overall O-Score on O-Bench (Beyond the Visible), with model average accuracy of 27.3% on occlusion rate estimation. For our Seen-From-There set, no reported result was found on Lacuna, so every number there comes from our reruns.

The strongest distillation competitor is a Spatial Forcing-style student. It uses the same backbone, aligns encoder tokens to a single-view VGGT teacher with a cosine loss, and has no feedback. A second version swaps in UniDepth v2 point-map targets, following the camera-aware model, which reports 68.35 on SPAR-Bench.

The strongest explicit competitor is a SymPL pipeline (Keep it SymPL). It extracts objects, depth, and orientation with off-the-shelf models and reprojects the scene into a symbolic top-down map. It reports 97.33% on COMFORT# "closer" and 91.50% on "facing" questions. We extend it to occlusion by adding the EUREKA amodal segmenter (Amodal Instance Segmentation with IRAIS Dataset) and an explicit ray cast from the agent marker through the estimated depth. This gives it a fair shot at visibility questions.

Three ablations isolate the proposal's ingredients. The first swaps the multi-view teacher for the single-view teacher while keeping the feedback, which isolates what the teacher contains. The second removes the feedback and gives the feedforward student extra transformer layers whose FLOPs match the feedback pass, then aligns those layers to the multi-view teacher, which isolates late modulation from added compute. The third replaces dense distillation with dense supervision on ray-cast visibility maps, which isolates teacher features from ground-truth labels.

Why not just…?

Why not just use Spatial Forcing with the multi-view teacher and skip the feedback pass? This is the most serious threat, and the proposal is built to settle it rather than argue around it. The depth- and FLOP-matched feedforward ablation receives exactly the same multi-view targets. Our prediction is that it matches the feedback model on visible camera-frame questions but lags on agent-frame occluded questions, because a single pass must commit to features before knowing whose viewpoint is asked about. If it does not lag, the contribution reduces to the privileged teacher, and we will report it that way.

Why not just run a symbolic pipeline like SymPL with an amodal segmenter bolted on? On unoccluded allocentric questions the pipeline will likely win. Its COMFORT# numbers are far above anything a learned student reports, so we do not claim otherwise there. Its weakness is that every stage commits to a hard estimate. A partially hidden object gets a wrong centroid from the detector, and monocular depth gives no surface behind the occluder for the ray cast to hit. We will measure accuracy as a function of the target's occluded fraction. The pipeline should degrade steeply above roughly half occlusion, while the distilled student should degrade gradually. If the curves do not separate, the explicit route is the better engineering answer.

Why not just fine-tune on more perspective-taking data, as MVSM does? The 3D-PC result is direct evidence that label-only fine-tuning reaches human parity in-distribution by learning 2D shortcuts, then drops to 66% when occluders break those shortcuts. Our retrained MVSM control on identical scenes will show whether this holds for our data. A counterfactual probe holds the target's image size and position fixed while moving only the occluder; a shortcut learner should be insensitive to that change.

Why not just supervise visibility directly from the meshes instead of distilling teacher features? We already have ray-cast ground truth, so this is a cheap and fair alternative, and it is included as an ablation. The argument for distillation is that a dense feature target forces the student to represent the hidden layout itself, not one bit per query. If dense visibility-map supervision matches distillation on the transfer benchmarks, then the teacher is unnecessary and the cheaper method is preferable.

Why not just scale the language model or prompt with chain of thought? O-Bench reports diminishing returns from scaling the language model from 7B to 72B and traces the bottleneck to the vision encoder. The Isle study found chain-of-thought helped some models and hurt others. We still report prompted baselines, but we do not expect them to be competitive.

Proposed Experiments

All learned methods train on the same ScanNet and Co3D-derived data. We evaluate on the in-domain Seen-From-There test scenes, then zero-shot on the 3D-PC (basic line-of-sight, depth order, and VPT-Strategy), Isle-Bricks and Isle-Dots, O-Bench, and ViewSpatial-Bench. The out-of-domain sets test whether the learned behaviour is geometry or dataset fitting. Metrics are accuracy for visibility and direction questions, absolute error for visible fraction, and the benchmarks' native scores elsewhere.

The first decisive check is the interaction between teacher and feedback. Both ingredients are predicted to matter only jointly. A multi-view teacher without feedback, or feedback trained against a single-view teacher, should each recover part of the gain on occluded agent-frame questions, and only the combination should approach the 87% human level on VPT-Strategy. A clear win would be exceeding the 66% fine-tuned reference by at least ten points.

The second check is selectivity. Splitting every benchmark by camera-frame versus agent-frame and by occluded fraction, the improvement over the matched feedforward student should grow with occlusion and vanish on visible camera-frame questions. If the improvement is uniform across all bins, that points to extra capacity and not to modulation.

The third check is a timing analysis. A linear probe for the depth of the hidden surface behind each occluder should decode poorly from pre-feedback tokens and well from post-feedback tokens. Shortcut sensitivity is tested with the counterfactual occluder-only perturbations. The contrast problem is tested by varying occluder contrast and checking that strong bottom-up edges do not erase the written-back completion.

Finally, we audit teacher failure, since distilled students inherit the teacher's blind spots. We stratify errors by scenes where VGGT's multi-view reconstruction disagrees with the mesh. If those scenes account for most of the student's failures, the next step is to supervise with mesh geometry rather than VGGT features.