TL;DR
Current evaluations of visual agents often conflate an agent's ability to plan where to look with its ability to interpret what it sees. This paper introduces budget-paired counterfactual audits, a protocol that isolates planning failures by comparing agent decisions across identical scenarios with varying observation budgets, revealing that most open-source models rely on fixed heuristics rather than true sequential planning.
What Problem Does This Paper Address?
When a Vision-Language Model (VLM) acts as an agent reading a dense document, it must decide which regions to "crop" or look at in high resolution, as it cannot process every pixel at once. This creates a sequential decision problem: every observation "spent" reduces the remaining budget.
Standard evaluation metrics usually just look at final answer accuracy. However, accuracy is a "noisy" signal that mixes three distinct things:
- Acquisition (Planning): Did the agent look at the right place?
- Adaptation: Did the agent change its strategy based on what it saw and how much budget remains?
- Terminal Decision: Given the evidence collected, did the agent reason correctly to find the answer?
Without separating these, we can't tell if a model is a "lucky" guesser or a sophisticated planner. For instance, a model might always look at the header first (a fixed heuristic). This works great if you have five reads, but it's a disaster if you only have one read and the answer is in a table at the bottom.
Methods
The authors propose two main innovations to "audit" these agents:
- Budget-Paired Counterfactuals: The researchers present the agent with the exact same state (same document, same question, same previous observations) but perturb only the remaining budget.
- Example: In a "routing" task, a header tells you whether the answer is in Table A or Table B. If you have two reads left, you should read the header first. If you have only one read left, reading the header is useless because you'll have no reads left to check the tables—you're better off just guessing Table A or B directly.
- Exact Trajectory Return Accounting: Using a formal task model, the authors calculate the exact continuation value (the maximum expected accuracy possible from a given state). This allows them to decompose the total loss in accuracy into stepwise acquisition loss (bad planning at a specific step) and terminal decision loss (bad answering at the end).
Datasets
The study utilizes two types of benchmarks:
- Controlled Reasoning Tasks: Synthetic "bit-return" tasks where the ground truth and optimal strategies are mathematically certain. These include "selector" and "branch" structures designed to test if models can adapt to moving value boundaries.
- FUNSD (Form Understanding in Noisy Scanned Documents): A real-world dataset of scanned forms. The authors introduced a "curated" vs. "original" reference scoring audit to show how sensitive VLM evaluations are to text normalization and ground-truth definitions.
Key Insights and Ideas
- The "First Read" Illusion: Many VLMs appear to be good planners because their first action is often optimal. However, the audit reveals this is usually just a "fixed-role heuristic" (e.g., "always read the route field first").
- Failure to Adapt: When the budget is reduced to a single read, open-source models fail to switch from "routing" to "direct answering." They keep looking at the "map" even when they no longer have time to follow it.
- Planning vs. Perception: Most of the accuracy loss in multi-step tasks occurs during the later stages of acquisition. Models are surprisingly good at the initial look but fail to use the information gained to inform their second and third looks.
- Scoring Sensitivity: On real-world documents, a model's "gain" over a baseline can fluctuate wildly (e.g., from 24.44 pp to 10.56 pp) just by changing how the reference text is normalized, highlighting a lack of "credible evaluation" in current leaderboards.
Key Results
- Total Planning Failure: All five tested open-weight VLMs (including Qwen3-VL and InternVL3.5) made zero optimal strategy switches across 40 paired anchor tests. They stuck to their fixed preferences regardless of the budget.
- The Loss Gap: In three-read sequences, models lost between 22.20 and 26.76 percentage points (pp) of expected accuracy during the second and third reads, despite the first read being near-optimal (losing less than 1 pp).
- Attainability: In contrast, GPT-5 mini (a hosted model) successfully made 48/48 optimal switches, proving that horizon-aware sequential planning is a reachable capability for current architectures.
Limitations and Future Work
- Declared Task Models: The "exact" part of the audit requires a formal model of the task (priors and read channels). While this is easy for synthetic tasks, creating these for open-ended, unstructured real-world scenes remains a challenge.
- Fixed Bindings: The current study provides the agent with "verified bindings" (it knows exactly where the regions are). Future work could integrate the localization problem—where the agent must first find the regions before it can decide which one to read.
- Model Scope: The audit focused on specific VLM families; testing a wider variety of architectures and "reasoning-heavy" models (like those with explicit CoT phases) would be a natural next step.
Summary updated