When to Zoom and When to Walk: Resolution-Calibrated Negative Evidence for Embodied Search

A robot looking for a small object has two ways to get a better look. It can crop and enlarge part of the camera frame it already has, which is a digital zoom that costs milliseconds. Or it can physically move closer or around an obstacle, which costs seconds and battery. Vision-language models (VLMs, systems that answer questions about images in text) make this choice harder than it looks. Their vision encoders shrink every frame to a few hundred pixels on a side. When the model fails to see a target, the failure could come from three places: the encoder threw away pixels the camera actually captured, the camera never captured enough pixels on the target, or the target was hidden behind something. Only the first is fixed by zooming. The others require walking.

This note proposes an embodied search agent that tells these cases apart using geometry it already has: depth, camera intrinsics, and the size of the region fed to the encoder. It turns each "I don't see it" into a belief update whose strength depends on how many pixels the target would have occupied and how visible that surface was. Empty close-up crops then count as strong evidence that the target is absent. Empty distant or occluded glances count for little. The same quantity tells the agent whether the next action should be a zoom or a move.

Core Claim

The active-zoom literature treats an empty high-resolution crop as reliable negative evidence. It is reliable only when the target would have been resolvable in that crop. In embodied settings this condition fails often and in predictable ways. We claim that a single measured function fixes this. The function is the backbone's detection probability as a function of the pixels a target would occupy after encoding, together with its visibility. That function is enough to calibrate negative evidence and to split resolution loss into an encoder part, fixable by zooming, and a sensor-or-occlusion part, fixable by moving. The result should be fewer physical moves at equal success, and fewer false "not here" conclusions, on small-object embodied search.

Formally, let $b(c)$ be the belief that the target lies in 3D map cell $c$. Let $\rho_c$ be the number of encoder pixels a typical target in $c$ would span in the current view, computed from depth, intrinsics, and crop scale. Let $v_c$ be the fraction of $c$ visible from the camera, computed from the depth buffer. After a view in which the VLM reports no target, the update is

$$b'(c) \propto b(c)\,\bigl(1 - p_d(\rho_c, v_c)\bigr),$$

where $p_d$ is the empirically measured probability that the backbone detects a present target at that pixel footprint and visibility. For each region the agent also computes $\rho^{\text{sensor}}_c$, the footprint if the region were cropped at the camera's native resolution. If $p_d(\rho^{\text{sensor}}_c, v_c)$ is high while $p_d(\rho_c, v_c)$ is low, the missing information is already in memory and a zoom is warranted. If both are low, only a move can raise the footprint or the visibility. Actions are chosen by expected reduction in belief entropy per unit time cost, where zooms and moves carry their measured wall-clock costs.

Relation to Prior Work

The closest work is FOVEA / The Perceptual Bandwidth Bottleneck in Vision-Language Models. It casts 2D cropping as sequential Bayesian experimental design (choosing each crop as the most informative next experiment) and shows that empty crops carry useful negative evidence. It works on single static images and assumes an ideal observer, meaning that a sharp crop guarantees the correct answer. It also suffers cold starts when its initial proposal is far from the target. We supply the method it lacks for this setting. The ideal-observer assumption is replaced by a measured $p_d$, the belief is lifted to a 3D map covering the whole scene, which removes the local-refinement cold start, and physical moves are added as a second kind of experiment. Its released code, iamlilAJ/active-vlm, gives us the 2D machinery to reuse.

ActiveVLA combines viewpoint selection and zoom for manipulation. It does so by re-rendering pre-reconstructed point clouds and scoring candidate cameras with hand-crafted visibility, distance, and diversity terms. It keeps no belief over where the target is and never asks whether a zoom would suffice. Its ablations are informative for us. Removing viewpoint selection drops RLBench success from 91.8% to 89.4%, and removing zoom drops it to 87.6%. Both mechanisms matter, but the paper offers no rule for choosing between them.

Uncertainty-Aware Gaussian Map for Vision-Language Navigation motivates the 3D belief. It builds a 3D Gaussian map with geometric, semantic, and appearance uncertainty that steers navigation. Its encoder tokens are fixed-resolution, so a small object seen from far away looks the same to it as an absent object. That is exactly the confusion we target.

Three RL-trained zoom systems constrain the design: DeepEyes, ACTIVE-o3, and ZoomEarth. All three report grounding drift or sparse-reward difficulty, and all three work only on 2D pixels. LOCI shows that VLMs rationalize wrong crops rather than reject them. It also reports that an oracle crop lifts V* accuracy from 83.8% to 98.2%, which says localization, not recognition, is the dominant error. Our $p_d$ serves as a geometric, non-linguistic critic that the model's own reasoning cannot talk its way past. AdaptiveNN motivates the cost framing: coarse-to-fine glimpses cut compute 4–8×. It does not model physical movement.

Research Program

The first component is measuring $p_d$. Using V* and HR-Bench images with known target boxes, together with Matterport3D renders with ground-truth object masks, we render each target at controlled pixel footprints and controlled partial occlusions. We then record the backbone's detection rate. The working hypothesis is that $p_d$ is a steep sigmoid in footprint for a given backbone. We also hypothesize that it transfers across domains well enough that one curve fitted on 2D photos works in indoor scenes. This hypothesis is testable, and if it fails, the method needs per-domain calibration and we will report that.

The second component is the embodied agent. It uses a fixed VLM backbone (Qwen2.5-VL-7B, matching DeepEyes) that answers "is the target in this view, and where?" It keeps a voxel belief map updated with the rule above, starting from a prior derived from the language instruction. It has an action planner that chooses among digital crops of any stored frame and navigation moves. We render at a native sensor resolution well above the encoder input, so that the encoder-versus-sensor split is real. The planner is training-free by design, so that the calibration itself is what gets tested.

The third component is a controlled evaluation split. We build small-object target-search episodes in Matterport3D scenes. Each episode is labeled by the target's native pixel footprint and occlusion at the start pose, so that results can be broken down by which kind of information loss dominates.

Baselines

The naive reference is a fixed-resolution agent that navigates and queries the VLM on full downsampled frames, with no zoom and no belief map. It shows how much the resolution bottleneck costs in embodied search at all.

The strongest existing embodied method is the Uncertainty-Aware Gaussian Map agent of Gao et al.. It reports 78% success on unseen R2R and a 3.94% gain in REVERIE remote grounding success over prior work. The file gives no absolute REVERIE value for it. For scale, the original REVERIE paper reports about 11.61% remote grounding success for its best model and 77.84% for humans. We run Gao et al.'s agent on our split and on REVERIE with the same backbone features where possible. Beating it means higher remote grounding success on the small-object split with no more path length.

The strongest zoom method is FOVEA ported to 3D. It uses the same voxel belief and the same action set as our agent, but the ideal-observer likelihood treats every empty crop as fully informative. In 2D, FOVEA reports a 77.7% mean on a Qwen3-30B backbone versus 75.1% for ReAct. This port is the ablation that isolates our key ingredient, the calibrated $p_d$.

Two further ablations complete the comparison. The first removes the zoom action, making the agent move-only, which is the ActiveVLA-style regime. The second removes the move-versus-zoom split and zooms first always.

All agents share the backbone, the native render resolution, a cap on VLM calls, and a wall-clock budget in which moves and zooms are charged their measured costs. Any tuned thresholds get the same number of validation trials for every method.

Why not just…?

Why not just use a higher-resolution encoder or tile every frame? Tiling at native resolution removes only the encoder share of the loss. It does nothing for targets that span too few sensor pixels or sit behind an occluder. It also multiplies tokens per frame, which Retrieval-Augmented Perception identifies as a context and latency problem. We include a tile-everything agent. We expect it to match us on the encoder-dominated slice of the split at several times the compute, and to lose on the sensor- and occlusion-dominated slices.

Why not just zoom first and walk only if the zoom fails? This is the real threat, because zooms are cheap. The cascade's weakness is in what "fails" means. With ideal-observer evidence, an empty zoom on a distant or partly hidden region wrongly clears it, which leads to misses and early stopping. Without negative evidence at all, the agent revisits the same region. The zoom-first ablation and the FOVEA port measure exactly this through false-absent rate and revisit rate. If they match our agent on success and path length, the contribution shrinks to the calibration study, and we will report it as such.

Why not just train end-to-end with RL over move and zoom actions? DeepEyes-style training could in principle learn when a glimpse is trustworthy. Prior work, however, reports sparse-reward difficulty (Active Exploring like a Pigeon) and grounding drift under answer-only rewards. We train a GRPO (group relative policy optimization, a reinforcement-learning method that ranks sampled attempts against each other) agent with the same action space and a matched episode budget. We then check whether its implicit behavior tracks $p_d$, for example whether it zooms more when the sensor footprint is ample but the encoder footprint is small. A match would validate the decomposition even if it weakens the case for an explicit model.

Proposed Experiments

The first check is whether $p_d$ can be measured and transferred. We fit the detection curve on 2D images and report its prediction error on Matterport3D renders. The rest of the proposal rests on this curve being steep and roughly domain-general.

The second check is calibration. We report expected calibration error for the agent's "target absent from this region" beliefs against ground truth, comparing $p_d$ with the ideal-observer likelihood. The ideal observer should be badly overconfident at small footprints and high occlusion.

The third check is the main comparison. On the small-object split and on REVERIE, we report remote grounding success, success weighted by path length, number of physical moves, number of zooms, and wall-clock time. Results are broken down by whether the start-pose loss was dominated by the encoder, the sensor, or occlusion. The predicted signature has three parts. On the encoder-dominated slice we should need far fewer moves than the move-only agent at equal success. On the occlusion slice we should beat zoom-first. On large, fully visible targets all methods should converge. That last result is a control: without it, any gain could not be attributed to the resolution mechanism.

The fourth check is a 2D sanity test. On V and HR-Bench, using the HR-Bench suite, we check that the calibrated likelihood does not hurt the FOVEA-style planner when moves are unavailable, and whether it reduces rationalized answers on wrong crops. DeepEyes' 90.1% on V is the reference for a 7B backbone.