Turning hours of raw footage into a short, query-specific summary is mostly a search problem. Given a request such as "show me every time I cooked something," a system has to guess a query, look at what comes back, notice what is missing, and ask again. Agents that refine their queries in this closed loop beat one-shot retrieval in text question answering. Training them with reinforcement learning (RL), meaning trial and error scored by a reward, is expensive in video, because every step of every trial decodes frames, runs a vision or audio model, or searches an index. The standard recipe, Group Relative Policy Optimization (GRPO), makes this worse. It samples 8 to 16 complete trajectories per training prompt and scores each one against the group average, so the expensive video calls are multiplied by the group size.
This note proposes to remove that multiplier. When we train on footage whose target summary is already annotated, we can give a separate scoring network, the critic, access to the answer key: the ground-truth set of shots. The agent itself never sees the key. With the key in hand, the critic can judge from a single trajectory how promising each intermediate query was. If that works, it delivers two savings at once. It replaces GRPO's group of samples, and it replaces the separate judge model that text agents use to score intermediate search steps. The question we test is whether, for a video query-refinement agent and at a fixed budget of video tool calls, one rollout with an answer-key critic trains a policy as good as 16-sample GRPO.
Core Claim
Text reasoning work has already shown that an actor-critic method can drop the group: Best Practice Critic Optimization (BPCO) matches 16-sample GRPO with one response per prompt. It does this by stabilizing the critic with bounded value outputs, Monte Carlo return targets, and privileged reference information. BPCO was tested only on single-turn math and rubric-graded text, where the cost of a rollout is token generation. Our claim is that the payoff is far larger in multi-turn video search, for two reasons. First, the cost of a rollout there is dominated by tool calls, and those scale linearly with the number of samples. Second, the privileged critic can absorb a role BPCO never needed: assigning credit to individual search turns.
To make the cost argument concrete, write $C \approx N \cdot T \cdot (c_{\text{gen}} + c_{\text{tool}})$ for the per-prompt rollout cost. Here $N$ is the number of sampled trajectories, $T$ the number of search turns, $c_{\text{gen}}$ the token-generation cost per turn, and $c_{\text{tool}}$ the per-turn cost of frame decoding, captioning, and index search. A critic adds a roughly constant forward/backward cost but sets $N=1$. When $c_{\text{tool}} \gg c_{\text{gen}}$, the saving approaches a factor of $N$, rather than the more modest saving seen in text.
Relation to Prior Work
Two systems motivate the gap directly. RIVER trains a 7B reasoning model with GRPO over a structured JSON description of the video. That description is built by SAM-2, OWLv2 and DepthAnything, and the model edits it in a single pass. FactGuard trains a video verification agent with GRPO to decide when to call web search and keyframe-inspection tools. Both pay the multi-sample cost on top of costly video perception, and neither reduces it. RIVER also cannot revise its query when perception fails.
Two other works supply the method. AgenticRag-R1 gives a text retrieval agent a last-in-first-out stack memory with a backtrack action, so failed searches can be popped instead of accumulating in the context. It reports +6.76% on HotpotQA with a 3B model. It scores intermediate queries with a separate general reward model, which is exactly the per-step overhead we want to eliminate. We reuse its action space and stack memory for video, and we replace its judge with the privileged critic. BPCO supplies the critic stabilization recipe. We extend it from single-turn text to multi-turn, tool-calling, multimodal episodes, and from a reference answer to a reference shot set.
Two works constrain the design. OmniCVR shows that vision-language retrieval models largely fail on queries that depend on sound: VLM2Vec reaches 12.4% R@1 on audio-centric queries versus 77.2% for an audio-aware model. The agent's clip index must therefore carry audio, or the search loop will fail regardless of how it is trained. OmniCVR also shows that descriptive audio text beats latent audio tokens (32.7% vs 13.6% R@1) but costs about 1.77x more processing time. That is precisely the kind of per-call cost that makes $c_{\text{tool}}$ large. LAVE shows that a prompted GPT-4 agent can plan retrieval queries over caption-based "visual narrations" of footage, but it requires human approval at every step and is never optimized. Finally, CLIP-It! is the strong single-pass alternative. It scores frames conditioned on the query text and reaches 54.55% F1 on QFVS. Any iterative agent has to beat it to justify the loop at all.
Research Program
The agent operates over a long video that has been pre-segmented into shots. Each shot is indexed by a visual embedding plus an audio caption, following OmniCVR's finding that text descriptions of audio work best. The agent's actions follow AgenticRag-R1, adapted to video. Search issues a text query to the shot index and pushes the top results onto the stack. Inspect decodes frames of one shot and returns a fresh caption; this is the expensive call. Backtrack pops an unhelpful result. Select commits a shot to the summary. Stop ends the episode. The terminal reward is the F1 overlap between the selected shots and the ground-truth summary, which is bounded in $[0,1]$. A bounded reward is exactly the setting where BPCO's arctangent value output applies cleanly.
The critic is a smaller copy of the policy backbone that sees the agent's stack together with the ground-truth shot set encoded as text. It predicts, after each turn, the final F1 the episode will reach. Following BPCO, it uses Monte Carlo targets, no advantage normalization, and length-adaptive advantage estimation across turns, with a bounded output head. The per-turn advantage it produces is our replacement for a process reward model. A query that adds nothing immediately but positions the next query well should receive credit, because the critic sees what remains to be found.
Annotated query-focused data is scarce: QFVS has only 46 queries. We therefore build training tasks automatically. We densely caption long egocentric and web videos, sample concept queries from the captions, and take the shots matching each concept as the ground truth. The real human-annotated benchmarks are reserved for evaluation. This makes label noise in the answer key a first-class risk, so we measure it directly (see the checks below).
Baselines
The naive reference point is the same backbone run zero-shot with the same tools and a fixed turn limit, prompted in the style of LAVE but without human approval. This shows how much training buys at all. The strongest non-iterative competitor is CLIP-It!, run in a single pass on the same shot segmentation. Its reported QFVS average F1 of 54.55% (CLIP-It!) is the number an iterative agent must exceed to justify the extra tool calls. For audio-dependent retrieval, the reference is AudioVLM2Vec's reported OmniCVR overall R@1 of 66.98% and audio-centric R@1 of 77.2% (OmniCVR).
The strongest RL competitor is GRPO with $N=16$ on the identical agent, tools, and synthetic training tasks, which is the recipe used by RIVER and FactGuard. We found no reported result on Lacuna for GRPO-trained query-refinement agents on QFVS, TVSum, or OmniCVR, so we train this baseline ourselves.
Fairness is enforced in two regimes. In the first, every method trains on the same number of prompts, which shows the raw cost gap. In the second, every method receives the same total budget of tool calls and GPU-hours, so GRPO sees roughly one-sixteenth as many prompts. Each method gets the same hyperparameter search budget of a fixed number of runs. We also run the AgenticRag-R1 recipe directly: GRPO plus a judge-model process reward. This isolates whether our critic substitutes for the judge.
Two ablations isolate the key ingredient. In the first, we keep the stabilized critic but remove the answer key, which is plain BPCO-style actor-critic. In the second, we keep the answer key but drop BPCO's stabilizers. The proposal counts as successful if, under matched tool-call budget, the answer-key critic reaches F1 at least equal to 16-sample GRPO on QFVS and TVSum. It should also beat CLIP-It!'s 54.55% on QFVS. And at matched prompt count, it should use at least roughly 8x fewer tool calls.
Why not just…?
Why not just run GRPO with a small group, say two or four samples? This is a serious threat, because a handful of samples may already provide a usable baseline. We train GRPO at $N \in \{2,4,8,16\}$ and plot final F1 against total tool calls. The proposal holds only if the one-rollout critic sits strictly above this curve. If $N=4$ matches it at similar cost, we will report that the critic is not worth its memory.
Why not just compute per-turn rewards directly from the ground truth, such as the recall gained by each search, and skip the critic? This is the strongest alternative. The answer key that feeds our critic could just as easily feed a hand-written per-turn reward. Our argument is that greedy recall gain is myopic: it rewards redundant hits and gives no credit to setup queries. The argument is also that single-rollout REINFORCE with shaped rewards still lacks a baseline, which leaves high variance. Neither point is settled by the file. We therefore run two direct comparisons: one-rollout REINFORCE with recall-gain shaping, and GRPO with the same shaping. If either matches the critic, the honest conclusion is that the answer key matters and the learned critic does not, and the paper will say so.
Why not just fine-tune on synthesized trajectories (supervised learning or rejection sampling) instead of RL? Because the ground truth is known, we could search offline for good query sequences and imitate them. FactGuard reports that supervised fine-tuning alone was insufficient to calibrate tool use. Getting More Juice Out of the SFT Data shows that folding reward learning into supervised training brings back generation and memory costs. Even so, we include an expert-iteration baseline given the same tool-call budget for trajectory search.
Why not just skip iteration and use a better single-pass scorer? CLIP-It! is that scorer. If our agent does not beat its 54.55% QFVS F1, the loop is unjustified. We also report F1 against average tool calls per query at test time, so that any accuracy gain is weighed against inference cost.
Decisive Checks
The central experiment trains all methods on the synthetic task pool from the same backbone. We evaluate on QFVS for query-focused summaries, on TVSum and SumMe (using category or title text as the query) for generic summaries, and on the audio-centric slice of OmniCVR, recast as a multi-turn search for the target clip. We report F1 or R@1 together with training tool calls, GPU-hours, and peak memory.
Three diagnostics decide whether the mechanism works for the stated reason. The first correlates the critic's per-turn advantages with a held-out estimate of each turn's true contribution, obtained by expensive multi-rollout continuations from that turn. If the critic is assigning credit to setup queries, it should beat greedy recall gain on this correlation. The second injects controlled label noise into the synthetic answer keys and measures how fast the critic's advantage degrades relative to GRPO. This is the risk specific to privileged information built from automatic captions. The third compares the audio-indexed agent with a vision-only index on OmniCVR's audio-centric queries. It confirms that gains come from refining queries rather than from the agent being unable to see the right shots.
The result we are betting on is that the answer-key critic matches 16-sample GRPO at about one-sixteenth the training tool calls, and that it beats both the recall-shaping and small-group alternatives under a matched budget. Losing to either of those would be a clear negative result with a specific cause.
References
- Best Practice Critic Optimization
- Text-Driven Reasoning Video Editing via Reinforcement Learning on Digital Twin Representations
- FactGuard: Agentic Video Misinformation Detection via Reinforcement Learning
- AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizing
- OmniCVR: A Benchmark for Omni-Composed Video Retrieval with Vision, Audio, and Text
- LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing
- CLIP-It! Language-Guided Video Summarization
- Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment