Choosing What to Observe: Budget-Paired Counterfactual Audits for Credible Evaluation of Visual Agents

Ruiqi Liu, Jinming Li, Mo Wang, Yang Lu, Yanlan KangarXiv 2026 · arXiv

TL;DR

Current evaluations of visual agents often conflate an agent's ability to plan where to look with its ability to interpret what it sees. This paper introduces budget-paired counterfactual audits, a protocol that isolates planning failures by comparing agent decisions across identical scenarios with varying observation budgets, revealing that most open-source models rely on fixed heuristics rather than true sequential planning.

What Problem Does This Paper Address?

When a Vision-Language Model (VLM) acts as an agent reading a dense document, it must decide which regions to "crop" or look at in high resolution, as it cannot process every pixel at once. This creates a sequential decision problem: every observation "spent" reduces the remaining budget.

Standard evaluation metrics usually just look at final answer accuracy. However, accuracy is a "noisy" signal that mixes three distinct things:

  1. Acquisition (Planning): Did the agent look at the right place?
  2. Adaptation: Did the agent change its strategy based on what it saw and how much budget remains?
  3. Terminal Decision: Given the evidence collected, did the agent reason correctly to find the answer?

Without separating these, we can't tell if a model is a "lucky" guesser or a sophisticated planner. For instance, a model might always look at the header first (a fixed heuristic). This works great if you have five reads, but it's a disaster if you only have one read and the answer is in a table at the bottom.

Methods

The authors propose two main innovations to "audit" these agents:

  • Budget-Paired Counterfactuals: The researchers present the agent with the exact same state (same document, same question, same previous observations) but perturb only the remaining budget.
  • Example: In a "routing" task, a header tells you whether the answer is in Table A or Table B. If you have two reads left, you should read the header first. If you have only one read left, reading the header is useless because you'll have no reads left to check the tables—you're better off just guessing Table A or B directly.
  • Exact Trajectory Return Accounting: Using a formal task model, the authors calculate the exact continuation value (the maximum expected accuracy possible from a given state). This allows them to decompose the total loss in accuracy into stepwise acquisition loss (bad planning at a specific step) and terminal decision loss (bad answering at the end).
Figure 1. Budget-paired counterfactual audit. (a) A form whose route field says which price table (A or B) holds the answer; numbered circles give the order of reads. With two reads left, a horizon-aware planner and a policy that always reads the route first make identical choices, so a single-budget test cannot separate them. With one read left, the route is worth less than either table (expected accuracy 0.50 versus 0.75), and only the planner switches. All five open-weight VLMs keep reading the route: 0/40 optimal switches. (b) Expected accuracy lost at each step of 160 three-read episodes per model: first reads are near-optimal (0.18–0.96 pp), while the second and third reads lose 22.20–26.76 pp.

Datasets

The study utilizes two types of benchmarks:

  1. Controlled Reasoning Tasks: Synthetic "bit-return" tasks where the ground truth and optimal strategies are mathematically certain. These include "selector" and "branch" structures designed to test if models can adapt to moving value boundaries.
  2. FUNSD (Form Understanding in Noisy Scanned Documents): A real-world dataset of scanned forms. The authors introduced a "curated" vs. "original" reference scoring audit to show how sensitive VLM evaluations are to text normalization and ground-truth definitions.

Key Insights and Ideas

  • The "First Read" Illusion: Many VLMs appear to be good planners because their first action is often optimal. However, the audit reveals this is usually just a "fixed-role heuristic" (e.g., "always read the route field first").
  • Failure to Adapt: When the budget is reduced to a single read, open-source models fail to switch from "routing" to "direct answering." They keep looking at the "map" even when they no longer have time to follow it.
  • Planning vs. Perception: Most of the accuracy loss in multi-step tasks occurs during the later stages of acquisition. Models are surprisingly good at the initial look but fail to use the information gained to inform their second and third looks.
  • Scoring Sensitivity: On real-world documents, a model's "gain" over a baseline can fluctuate wildly (e.g., from 24.44 pp to 10.56 pp) just by changing how the reference text is normalized, highlighting a lack of "credible evaluation" in current leaderboards.

Key Results

  • Total Planning Failure: All five tested open-weight VLMs (including Qwen3-VL and InternVL3.5) made zero optimal strategy switches across 40 paired anchor tests. They stuck to their fixed preferences regardless of the budget.
  • The Loss Gap: In three-read sequences, models lost between 22.20 and 26.76 percentage points (pp) of expected accuracy during the second and third reads, despite the first read being near-optimal (losing less than 1 pp).
  • Attainability: In contrast, GPT-5 mini (a hosted model) successfully made 48/48 optimal switches, proving that horizon-aware sequential planning is a reachable capability for current architectures.
Figure 3. Breakdown of accuracy loss across a three-read trajectory for different models.

Limitations and Future Work

  • Declared Task Models: The "exact" part of the audit requires a formal model of the task (priors and read channels). While this is easy for synthetic tasks, creating these for open-ended, unstructured real-world scenes remains a challenge.
  • Fixed Bindings: The current study provides the agent with "verified bindings" (it knows exactly where the regions are). Future work could integrate the localization problem—where the agent must first find the regions before it can decide which one to read.
  • Model Scope: The audit focused on specific VLM families; testing a wider variety of architectures and "reasoning-heavy" models (like those with explicit CoT phases) would be a natural next step.

Summary updated

Abstract

Visual agents that read dense documents must ration a finite observation budget across candidate regions, yet answer accuracy conflates where an agent looks, how it adapts to the reads that remain, and how it an?swers once observation stops. We introduce budget-paired counterfactual audits, an evaluation protocol that holds public evidence and region bindings fixed while perturbing only the remaining budget or de?clared task parameters, and credits every acquired history with exact continuation values. The resulting accounting decomposes lost expected accuracy into stepwise acquisition loss and terminal decision loss, separating planning from fixed-role heuristics. Across forty paired routing anchors, five open-weight
BibTeX
@misc{liu2026choosing,
  author = {Ruiqi Liu and Jinming Li and Mo Wang and Yang Lu and Yanlan Kang},
  title = {{Choosing What to Observe: Budget-Paired Counterfactual Audits for Credible Evaluation of Visual Agents}},
  year = {2026},
  eprint = {2611.24867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24867},
}
Figure