Profiling Wasted Perception to Refine Video Query Programs

Long-video question answering wastes most of its compute. A system decodes, embeds, detects and captions frames that turn out to have no bearing on the answer. Today that waste is cut in one of two ways. Researchers either train a neural frame selector, or they hand-write a search procedure that decides which frames to inspect. A third option is available. A language model writes the frame-inspection procedure as ordinary Python that calls vision tools. The program runs on a development set of videos, and a profiler reports back which tool calls actually changed answers and which were wasted. The model then rewrites the program to stop paying for the wasted ones. This is the loop that code-optimization systems already use with CPU profilers. Here the "hot spots" are perceptual rather than computational.

The core bet is that a video-specific profiler is what lets iterative code refinement cut perceptual cost without losing accuracy. That profiler measures how much each tool call influences the final answer, not just its runtime. The refinement happens offline. What ships is a compact, deterministic program with no language-model loop at inference time. That distinguishes it from agents that deliberate over every question.

Core Claim

Existing work sits on either side of this idea without joining it. ViperGPT has a language model write Python that composes vision models and loops over video frames. It runs open-loop: it never executes, inspects or repairs its own programs, and it iterates over frames uniformly. Logic-in-Frames (VSLS) shows that a structured search over frames can reach 41.4% temporal coverage of ground-truth frames while sampling 1.4% of them. That search is hand-engineered and fixed, and it pays roughly 7.8 seconds of detector overhead per query. AVUA refines its sampling through verbal self-reflection inside a per-question ReAct loop, which is expensive and nondeterministic.

On the code side, two systems use execution profiles as refinement feedback. POLO uses runtime profiles and a PageRank-style ranking of functions to direct rewrites of C/C++ bottlenecks. ICER does the same with runtime and memory profiles on competitive-programming problems. Neither has any notion that a function call on frame 3,412 might be wasted because its output never reaches the answer.

The claim is specific. We would build a profiler that attributes answer influence to individual perceptual calls in a video program, and feed its output into an ICER-style refinement loop that accepts candidates under a relaxed rule and validates them on held-out videos. The resulting programs should reach a better accuracy-versus-total-compute tradeoff than both learned selectors and hand-written search. Total compute here counts the selection overhead itself. The gain must also come from the influence signal, not from generic retry-and-debug or from tuning thresholds.

Research Program

The execution environment. The environment exposes a small, cached API over a video:

  • frame decoding at a chosen timestamp;
  • CLIP-style text–frame similarity over batches;
  • open-vocabulary detection on frame grids, reusing the grid trick from VSLS;
  • a cheap captioner;
  • a final call to a fixed answering VLM with a chosen frame set.

Every call is logged with its measured floating-point cost, wall-clock time, cache status and returned value. The answering VLM is held fixed across all methods, so any difference comes from which frames reach it and at what upstream cost.

The influence profiler. This is the new component. For a program $P$ run on development example $i$, let $c$ index an individual perceptual call. We define its influence as

$$I(c) = \frac{1}{|D|}\sum_{i \in D} \big|\, m_i(P) - m_i(P_{\setminus c}) \,\big|,$$

where $D$ is the development set, $m_i(P)$ is the answering model's margin on the correct option under the full program, and $P_{\setminus c}$ re-executes the program with call $c$'s output replaced by a null value, so the program's downstream control flow reacts. Replaying individual calls is affordable offline because every perceptual output is cached. Only the control flow and the final VLM call need rerunning, and we sample calls rather than enumerating all of them.

Influence is then aggregated to source lines and loop iterations. It is paired with each line's share of total compute, which gives a ranked report. For example: "line 14, the dense CLIP scan over all frames, uses 61% of compute and contributes 4% of answer influence; line 22, detector verification on top-8 frames, uses 9% and contributes 55%." This is the analogue of POLO's function ranking, but the ranking is by wasted perception rather than by time alone.

The refinement loop. It starts from a seed program. The seed is either a ViperGPT-style uniform loop or a transcription of the VSLS search. Each round, a code model receives four things: the current program, execution errors and tracebacks, the influence-versus-cost report, and aggregate accuracy and compute on the development split. It proposes edits.

Following ICER's finding that strict monotonic acceptance causes "efficiency overfitting" to noisy small test sets, we accept any candidate that runs without error and stays within a tolerance of the best development accuracy. The final selection among accepted candidates is made on a separate validation split of disjoint videos. Following the rollback design of Fang et al., a candidate that regresses validation accuracy beyond tolerance reverts to the last accepted version.

We deliberately do not ask the model to label query complexity or temporal difficulty up front. ICER reports that frontier models reach only about 60% accuracy at explicit Big-O labeling, and that forcing such predictions degrades optimization. We expect the same risk for predicting how many frames a question needs. The program may branch on the question text if the model chooses, but budget allocation should come from measured influence.

The output is one program per training distribution, with ordinary query-conditioned branches. At test time it runs with no language-model calls except the final answer.

Baselines

Do-nothing reference. Uniform sampling at 8, 16 and 32 frames into the same answering VLM. On NExT-QA, Flexible Frame Selection reports 61.6% for a 32-frame uniform baseline. The M-LLM frame selection paper reports 68.1% with 8 uniform frames in its setting. We rerun uniform sampling ourselves with our answering model rather than borrow those numbers.

Strongest learned selector. The Qwen2.5-1.5B scout of Hu et al.. It uses 9-token-per-frame pooling, a learned score token and non-maximum suppression. It reports raising Qwen2-VL 7B on LongVideoBench from 53.3% to 57.0% with 32 selected frames, and 68.5% on NExT-QA with 4 selected frames versus 68.1% for 8 uniform ones. This is the most direct competitor: training-based, cheap at inference, and evaluated with an open answering model we can match.

We also run FFS, which reports 62.2% versus 61.6% on NExT-QA at a 13.5x downstream compute reduction. Both use their released recipes and the same answering model. We charge them the scout's own forward-pass cost.

Strongest search baseline. VSLS from Logic-in-Frames, run with its published settings and charged its detector overhead. It reports an 8.7% accuracy gain for GPT-4o on LongVideoBench.

Strongest per-query agentic baseline. AVUA, which reports 66.98% on EgoSchema while accessing about 93% fewer frames. We also report VAP at 68.1% on EgoSchema and 81.4% on NExT-QA as reference ceilings. Their answering models differ from ours, so we rerun them with our answering model where code permits. Otherwise we compare only on total frames touched and wall-clock time.

Ablations isolating the key ingredient. The first keeps the identical loop but gives only tracebacks plus scalar accuracy and total cost, with no influence report. The second is a self-debug-only loop that repairs crashes and stops. The third is a strict monotonic acceptance rule in place of the relaxed one.

All refinement variants get the same code model, the same number of candidate executions and the same development and validation splits.

What counts as beating a baseline. Strictly higher accuracy at equal or lower total compute, including selection overhead. Or equal accuracy, within the seed-to-seed standard deviation, at no more than half the compute of Hu et al. The comparison is on held-out test videos, averaged over three refinement seeds.

Why not just…?

Why not just train a better learned selector? This is the main threat on in-distribution benchmarks. Hu et al. and FFS may well match us there. Our case rests on three things a trained selector does not give. First, the compute accounting includes a 1.5B-parameter scout pass over many candidate frames, which our programs can avoid by scanning cheaply first. Second, programs can express compositional checks, such as ordering or the co-occurrence of two objects; VSLS argues that semantic-similarity selectors miss exactly these. Third, a program can be adapted to a new benchmark with a few hundred labeled development examples instead of pseudo-label pretraining. The cross-benchmark transfer experiment below settles this. If the learned selector wins there too, the proposal's value reduces to interpretability, and we will report that plainly.

Why not just tune the knobs of VSLS with a black-box optimizer? Much of what a code model "discovers" might be threshold and budget tuning. We therefore give Bayesian optimization the same number of evaluations over every numeric parameter of the VSLS program: grid size, detection thresholds, diffusion rate and frame budget. If this matches refined programs, then structural rewrites add nothing. We will then inspect the diffs of the winning refined programs to see whether their gains come from changed control flow, such as early exits, reordered cheap-before-expensive calls or removed scans, or from constants alone.

Why not just let an agent reason per question, as AVUA does? Per-question agents can adapt to each video. They pay repeated language-model and tool calls every time, and their behavior varies from run to run. We report wall-clock time and cost per question alongside accuracy. We also report an oracle-style comparison: AVUA's accuracy against our program's accuracy at matched latency.

Why not just add error-message retries to ViperGPT? Retries fix crashes, but they do not remove wasted perception. The self-debug-only ablation measures exactly this gap. Iterative refinement is known to plateau after a few rounds (Yan et al.), so we also track whether the influence report extends useful rounds beyond where scalar feedback stalls.

Proposed Experiments

Main evaluation. We refine on the training portions of NExT-QA and LongVideoBench and evaluate on held-out test videos. The answering models are Qwen2-VL 7B and a larger open VLM. We report accuracy, frames decoded, total floating-point operations including selection, and wall-clock time per question. We include accuracy-versus-compute curves obtained by sweeping a cost penalty in the acceptance rule. On LV-Haystack we measure keyframe retrieval directly, so that selection quality is assessed separately from answer generation.

Transfer. Programs refined on NExT-QA are evaluated unchanged on EgoSchema. EgoSchema's median temporal certificate of about 100 seconds (EgoSchema) stresses whether programs that learned to skip also learned when not to skip. We then allow 200 EgoSchema development examples of re-refinement and compare against fine-tuning the Hu et al. selector on the same examples.

Decisive checks. Three results would settle the proposal. The full loop must beat the scalar-feedback ablation on the compute–accuracy frontier at matched evaluation budget. If it does not, the influence profiler is not doing the work. Refined programs must beat the black-box-tuned VSLS. Relaxed acceptance with held-out selection must show a smaller gap between development and test accuracy than strict monotonic acceptance, mirroring ICER's efficiency-overfitting result in the perceptual setting.

Robustness check. Because POISONVID shows that prompt-guided samplers can be steered away from informative frames, we will test whether refined programs, which can include fixed coverage guarantees, degrade less than the Hu et al. selector under the same perturbations. This test is conditional on their attack being reproducible from the paper.