Long videos contain far more frames than a vision-language model can read. The strongest current systems therefore pick frames adaptively at answer time. An agent looks at a few frames, decides what is missing, and asks for more. It works: FrameThinker reaches 76.1% on LongVideo-Reason while reading about 20.6 frames. The cost is that every question now involves several rounds of model calls and tool calls, and VideoAgent2 reports heavy latency from exactly this loop. This proposal asks whether that adaptivity can be moved out of answer time. The idea is to write the frame-selection policy as an ordinary Python program that runs cheaply per question, and to improve that program offline with the same execution-feedback loop that code models use to fix failing programs.
The missing piece is the feedback. Code repair works because a failing test says where the program went wrong. A frame sampler that gets a question wrong only yields a wrong answer. We build the equivalent of a failing-test report for frame sampling, which we call an evidence-miss trace. For a failed question, an offline probe finds which stretch of the video, once added to the program's chosen frames, makes the answering model correct. The trace then records what cheap signals the program could see at that stretch and why its logic skipped it. The signals include text-image similarity rank, frame-difference magnitude, shot boundaries, and position in the video. A code model reads batches of these traces plus profiler numbers and edits the sampler. Regressions on held-out questions are rolled back.
Core Claim
We claim that a sampler program refined against evidence-miss traces recovers most of the accuracy gain that multi-turn sampling agents get over uniform sampling. It should do so at near-uniform cost: a single call to the answering model, and no extra language-model rounds at inference. We further claim that the traces are the essential ingredient. Refining the same program against scalar accuracy alone, or against tuned hyperparameters of a fixed hybrid sampler, should fall clearly short. The object being refined is new. Prior video systems refine a per-question program built from a fixed module library (VideoPro), train a policy over fixed actions (FrameThinker, VideoBrain), or generate a program once with no repair (ViperGPT). None of them produces a reusable sampling algorithm debugged across a training distribution.
Research Program
The sampler is a Python function with a fixed signature. It takes three inputs: the question text, a frame budget $B$, and a table of cheap per-frame features computed once per video. These features are CLIP similarity to the question and to each answer option, frame-difference energy, shot-boundary flags, and timestamps. It returns at most $B$ frame indices. It may use arbitrary control flow, such as bisection around a similarity peak, per-shot quotas, or looking again after an option-specific match. It may not call the answering model. The target is
$$\max_{\pi}\ \mathbb{E}_{(v,q,a)}\big[\mathbf{1}\{\mathrm{VLM}(q, v[\pi(q,v,B)]) = a\}\big] \quad \text{subject to} \quad |\pi(q,v,B)| \le B,$$
where $\pi$ is the sampler program, $v$ a video, $q$ a question with answer $a$, and $v[\cdot]$ the selected frames. The answering model is Qwen3-VL-8B, held fixed. Both VideoBrain and VideoPro build on this model, which keeps comparisons honest.
The refinement loop borrows its structure from the code-refinement literature, and each piece plays a specific role. OpenCodeInterpreter shows that execution feedback lifts a code model beyond its single-turn output (HumanEval Pass@1 from 79.3 to 82.9). It motivates the loop. POLO rejects candidate rewrites using measured runtime rather than assumed cost. It supplies the profiler component: every candidate sampler is timed for feature lookups and decoded frames. LearnAct grows and repairs a library of Python actions from failure cases, raising AlfWorld success from 54.7% to 72.2%. It supplies the precedent of refining reusable code across tasks rather than per instance. We let the refiner add helper functions, such as a per-shot quota routine, that persist across rounds.
Two negative results constrain the design. Yan et al. find that feedback-driven gains plateau after a few rounds. Fang et al. find that edits often introduce regressions. We therefore keep a small population of candidate programs, restart from the best when progress stalls, and accept an edit only if it does not lose accuracy on a held-out validation slice.
Evidence-miss traces are built as follows. For a sampled subset of failed training questions, the video is split into fixed windows. Each window is appended in turn to the program's selection, and we record the windows that flip the answer to correct. Clue annotations are used where the training data provides them; VideoPro trains from CG-Bench, which we use as the refinement pool. The trace for each flipping window lists the cheap-feature values there, the rank those values held under the program's own scoring, and the line of program logic that excluded it. The refiner sees roughly fifty such traces per round, grouped by question type, alongside aggregate accuracy and profiler output.
This probing is expensive, but it is paid once, offline. We report its cost in answering-model calls so the amortization can be judged.
Baselines
The naive reference point is uniform sampling at the same budget $B$ with the same answering model. Every gain is measured against it. We use budgets of 16, 32, and 64 frames to trace a curve rather than a single point.
The second baseline is CLIP top-$k$ retrieval, which ranks frames by similarity to the question and takes the top $B$. It is the cheapest query-aware sampler. VideoBrain attributes 54.3% of its errors to imprecise retrieval-based sampling, which suggests it leaves headroom.
The third baseline is the one we most need to beat among cheap methods. It is a hand-designed hybrid that mixes uniform and top-$k$ frames, with a temporal spread penalty and a per-shot cap. Its hyperparameters are grid-searched on the same refinement pool, and the search gets the same number of answering-model evaluations as our refinement loop, probing calls included.
Among existing adaptive methods, the strongest single-call competitor is VideoBrain. It reports +9.0% on LVBench and +8.1% on LongVideoBench over uniform sampling with Qwen3-VL-8B, using 30–40% fewer frames. The file gives no absolute LVBench accuracy for it, so we compare against its reported gain over uniform, re-measured in our setup if the code is available.
The strongest multi-turn competitors are VideoPro and FrameThinker. VideoPro reports over 10% gain on LVBench and a 6.7% average gain for Qwen3-VL-8B. FrameThinker reports 76.1% on LongVideo-Reason with 20.6 frames, versus 72.0% for LongVILA-R1 with 512 frames. That 76.1% is the headline number to approach on LongVideo-Reason. We do not expect to beat these agents on accuracy. We expect to dominate them on accuracy per unit of wall-clock time, and we report both.
Three ablations isolate the key ingredient. The first refines against scalar accuracy only, with no traces. The second removes rollback. The third takes the initial program with no refinement, which is the ViperGPT-style one-shot condition.
The proposal beats the field if it meets two conditions on held-out benchmarks never seen during refinement. The refined program must exceed the tuned hybrid by at least two points at equal budget. It must also recover at least half of VideoBrain's reported gain over uniform, with one model call per question.
Why not just…?
Why not just grid-search a CLIP-plus-uniform hybrid? This is the most serious threat. Most of what a refined program learns might be "mix similarity peaks with coverage," which a tuned hybrid already encodes. The equal-budget hybrid baseline exists to settle this. If the refined program cannot beat it by a clear margin on held-out benchmarks, the claim fails. We would then report that code-level refinement adds nothing over parameter search for frame selection. A secondary check is to read the refined programs themselves. If they only contain retuned constants, the result is the same.
Why not just train a learned selector, as in the Flexible Frame Selector or VideoBrain? Buch trains a lightweight module that cuts compute up to 13.5x, and VideoBrain uses reinforcement learning over two tools. A trained selector may well match or beat a program in-distribution. Our bet is narrower. A program is inspectable, needs no gradient training of any model, and may transfer better across benchmarks because its logic is explicit. The cross-benchmark transfer experiment below tests exactly this. If the learned selectors transfer as well, the remaining advantages are interpretability and cost, and we will say so.
Why not just refine per question, as VideoPro and FrameThinker do? Per-question loops see the actual frames and can react to them, which an offline program cannot. They will likely stay more accurate on the hardest temporal-reasoning questions. The price is several language-model and tool rounds per question. We therefore compare on an accuracy-versus-latency curve, not accuracy alone. We also report the per-question-type breakdown on LVBench's six capability categories to show where offline programs are enough and where they are not.
Why not just feed hundreds of uniform frames into a long-context model? The file's evidence argues against this. FrameThinker's 20.6 selected frames beat 512 uniform frames. EgoSchema reports that some models degrade as more frames are added. LVBench reports that even the top model reaches only 67.4% against roughly 94% for humans. We still include a 256-frame uniform run as a ceiling check at high cost.
Proposed Experiments
Refinement runs on CG-Bench. Evaluation is on LVBench, LongVideoBench, the long subset of Video-MME, MLVU, and EgoSchema, all held out, plus LongVideo-Reason for direct comparison with FrameThinker. Metrics are accuracy, frames used, answering-model calls per question, and measured wall-clock time per question including feature extraction. We also report the one-time offline refinement cost in model calls.
The first decisive check is the trace ablation. At equal refinement budget, trace-driven refinement must beat scalar-accuracy refinement on held-out benchmarks. Otherwise the traces add nothing, and the proposal reduces to program search.
The second is transfer. A program refined only on CG-Bench is scored on the five other benchmarks. Its held-out drop is compared to the equal-budget hybrid and to a learned selector trained on the same pool. This tests whether code-level refinement overfits benchmark quirks.
The third is the Pareto comparison with VideoPro and FrameThinker. It shows whether the program sits on the accuracy-versus-latency frontier or merely below it.
We will also track refinement curves over rounds. They should show whether the plateau reported by Yan et al. appears and whether restarts plus rollback delay it. For a qualitative audit, we will categorize the final programs' logic against the failure types VideoBrain reports: intervals chosen too broadly, and queries that are too vague.
References
- FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting
- VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoT
- VideoPro: Adaptive Program Reasoning for Long Video Understanding
- VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding
- ViperGPT: Visual Inference via Python Execution for Reasoning
- OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
- POLO: An LLM-Powered Project-Level Code Performance Optimization Framework
- Empowering Large Language Model Agents through Action Learning
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- MEIC: Re-thinking RTL Debug Automation using LLMs
- Flexible Frame Selection for Efficient Video Reasoning
- EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding
- LVBench: An Extreme Long Video Understanding Benchmark