Vision-language models that "think with images" can zoom into or crop a picture mid-answer, and they now post large gains on high-resolution benchmarks. These benchmarks reward only the final answer. A higher score could therefore mean three different things. The model may have learned to look at the right region. It may reason better once it has looked. Or the evaluation script may have changed in ways that happen to favor it, such as how much padding surrounds a crop, which image-resizing filter is used, or how many zoom turns are allowed. This proposal builds a locked, containerized evaluation harness for multi-turn zoom-and-crop agents. Inside it, we intervene on the evidence the agent receives: we replace its own crops with ground-truth crops, with blank crops, or with misleading crops, and measure how the answer changes. The intervention splits every error into a perception share (the agent looked in the wrong place) and a reasoning share (the agent had the right evidence and still failed). It also measures how much of each method's reported gain survives a fixed protocol.
This matters because the field is about to make a large bet on active perception. The bet rests on gains measured with custom scripts. Those gains may be partly protocol artifacts, and active perception may not help at all on the reasoning-heavy categories where models are weakest.
Core Claim
We claim, and will test, two things. First, under a fixed protocol with matched visual-token budgets, a substantial part of the spread between published active-perception methods comes from evaluation choices rather than the methods themselves. We expect at least some pairwise rankings to flip. Second, swapping in ground-truth evidence will show that active perception closes most of the perception share of error on search-style questions. It will close little of the reasoning share. On categories such as perspective transformation, models fail even when handed the correct crop.
If the first claim holds, comparisons in this area need a shared harness. If the second holds, the "ideal observer" assumption behind sequential crop search is falsified on exactly the questions that matter most. Under that assumption, a sharp enough crop guarantees a correct answer.
Grounding in Prior Work
Several papers motivate the attribution question. The Perceptual Bandwidth Bottleneck in Vision-Language Models (FOVEA) treats crop selection as sequential experimental design, choosing views that are expected to be most informative. It reports a 77.7% mean across HR-Bench, MME-RealWorld-Lite, V* Bench and CV-Bench. It also concedes that models hallucinate even after resolving the right patch. MATH-Vision manually splits GPT-4V errors into 31.9% recognition and 42.2% reasoning. GeoSQA shows that models stay near chance even when diagrams are given as perfect text. These results suggest the reasoning share is large. None of them measures it causally inside multi-turn trajectories.
Three methods supply the systems we evaluate, and all three report results from their own pipelines. DeepEyes uses reinforcement learning to train a 7B model to call a zoom tool, and documents both "zoom-happy" over-querying and drift into wrong regions. SCOPE directs crops toward the points where competing symbolic hypotheses disagree. TreeVGR trains a model to emit bounding boxes alongside its reasoning.
TreeBench, from the same TreeVGR paper, is the closest existing evaluation. It scores whether predicted boxes overlap ground truth, but only for single-pass outputs, and box overlap is correlational. A model can draw the right box and ignore it. Our interventions test whether the evidence actually changed the answer. LEDGERMIND names failure types such as "phantom grounding," where a claim cites evidence that does not support it. It diagnoses these with deterministic text checks rather than counterfactual evidence swaps. TIR-Bench explicitly fails to separate tool failure from reasoning failure. GuessBench measures a passive-to-active collapse but restricts the agent to yes/no questions.
The methodology comes from standardized-evaluation work. NerfBaselines showed that a trivial protocol choice (background color) flips leaderboard ranks, and that 12 of 19 codebases no longer install. BARS showed that equal tuning budgets erase claimed gains. We carry both lessons into a multi-turn setting where the model's actions change its own inputs, which those frameworks do not support.
Research Program
The harness wraps each agent in a container. It owns every image operation, so the agent emits only crop coordinates and the harness performs the crop, padding, resizing filter, re-encoding resolution and turn limit. It also logs every crop and the visual tokens it consumed. Prompt templates and answer parsing are fixed per benchmark. We build on VLMEvalKit, which already hosts TreeBench, and add the multi-turn tool loop it lacks.
Attribution uses replay with intervention on TreeBench, whose 405 questions carry boxes for every referenced object. For each question we record three accuracies:
- $A_{\text{pol}}$: the agent runs freely.
- $A_{\text{orc}}$: each crop the agent requests is replaced by the ground-truth crop.
- $A_{\text{none}}$: crops return the downsampled full image only.
From these we define
$$\text{perception share} = A_{\text{orc}} - A_{\text{pol}}, \qquad \text{reasoning share} = 1 - A_{\text{orc}},$$
with all quantities computed per competency category. A fourth condition substitutes a plausible distractor crop. If the answer does not change, the agent is ignoring its evidence, which is a direct behavioral test of phantom grounding.
As a secondary probe, we run the same agents on Chart101 with chart titles kept, deleted, or made misleading. The question is whether zooming reduces reliance on textual shortcuts or only gives the model new material to rationalize them.
Baselines
Naive reference points. The do-nothing reference is the base Qwen2.5-VL-7B in single-pass mode. The file implies roughly 37.0 on TreeBench (TreeVGR's 50.4 minus its reported +13.4) and reports 71.2% on V* Bench. A second naive reference is static tiling: the image is cut into a fixed grid and each tile is encoded, with the visual-token count matched to each active agent's measured mean. This is the fair version of the objection that "just use more resolution" would do as well.
Strongest existing methods. We include the strongest published methods, all re-run inside the harness from released checkpoints or code with their published settings. These are DeepEyes (reported 90.1% on V* Bench), TreeVGR-7B (reported 50.4 on TreeBench), SCOPE on Qwen3-VL-8B (reported 52.3% on TreeBench, up from 46.4%), and FOVEA with a Qwen3 backbone (reported 77.7% mean). OpenAI-o3's reported 54.87 on TreeBench serves as a proprietary ceiling, where API access permits.
We do not retune anyone's training. We give every method the same harness-level settings: padding, resizing filter, turn cap, and token budget. Training-free methods such as FOVEA and SCOPE get a fixed, equal number of hyperparameter configurations on a held-out split, following the equal-tuning lesson from BARS.
What counts as beating a baseline. "Beating" here is not a new accuracy record. The harness succeeds if its decomposition predicts something that answer-only scoring cannot. The test is whether the perception share measured on a held-out half of TreeBench predicts which method gains more from a fixed increase in zoom budget on the other half. A second test is whether intervention-based attribution agrees with a small human error audit better than TreeBench's box overlap does.
Ablation of the key ingredient. The key ingredient is the oracle substitution. Removing it reduces the harness to answer accuracy plus box overlap, which is the TreeBench-style comparison.
Why not just…?
Why not just use TreeBench's box-overlap score as the perception metric? Box overlap records where the model pointed, not whether that evidence drove the answer. A model can localize correctly and still answer from a textual prior, and the distractor-crop condition detects exactly that case. We test this directly. If box overlap correlates as well as the intervention measure with the human audit and with budget-gain prediction, the intervention adds nothing, and we will report that.
Why not just run everything in VLMEvalKit as is? This is a real partial threat, because shared scripts already remove some drift. They do not, however, own crop padding, the resizing filter, or the turn cap inside a tool loop, and NerfBaselines shows that choices of this size can flip rankings. The protocol sweep settles the matter. If rankings are stable across it, a plain shared toolkit suffices, and the standardization contribution shrinks to the attribution layer.
Why not just give the model higher-resolution static tiles? The reported evidence favors active search: DeepEyes improves HR-Bench 4K and 8K by 6.3 and 7.3 points, and ACTIVE-O3 lifts small-object detection on SODA-A from 0.7 to 9.2 AP. Those comparisons were not matched on tokens. Our matched-token tiling baseline answers this cleanly, and if tiling ties active agents at equal tokens, that is itself a headline finding.
Why not just do a manual error audit like MATH-Vision's? Manual labels do not scale across methods, protocol settings and replays, and they cannot answer counterfactual questions. We still run a 100-question human audit as the validation target for the automated decomposition, rather than treating it as a competitor to it.
Proposed Experiments
The first experiment is the protocol sweep. We use a factorial grid over crop padding (three levels), resizing filter (bilinear, bicubic, Lanczos), turn cap (2, 4, 8) and re-encoding resolution (two levels). It runs on V* Bench, HR-Bench 4K/8K, MME-RealWorld-Lite and TreeBench. We report the spread of each method's score, and Kendall's tau between method rankings across settings, with seed variance as the noise floor. A tau that stays well below 1 beyond that floor supports the first claim. A tau near 1 refutes it.
The second experiment is the attribution study on TreeBench described above, reported per category with bootstrap intervals, given the small benchmark size. The decisive check is the perspective-transform category, where TreeVGR scores 22.35%. If oracle crops leave this category near its current level while lifting object retrieval and OCR close to ceiling, the ideal-observer assumption fails where it matters. If oracle crops fix perspective transform too, then perception was the bottleneck after all, and further investment in search is justified.
The third experiment reruns the agents under the Chart101 title ablation to test whether active perception reduces reliance on textual shortcuts.
Throughout, every result is reported as accuracy against visual tokens consumed, so no method gains from spending more compute unseen.
References
- The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
- SCOPE: Symbolic Constraint-Guided Active Perception
- Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Method
- LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
- TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
- When Seeing Is not Enough: Revealing the Limits of Active Reasoning in MLLMs
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- BARS: Towards Open Benchmarking for Recommender Systems
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details