From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents

Can Zhang, Baofeng Zhang, Xiaotian Han, Junyuan Shang, Yuchen Ding, Shuohuan Wang, Dianhai Yu, Ruirui LiarXiv 2026 · arXiv

TL;DR

VESTA is a training-free long-video AI agent that dynamically adapts its evidence-gathering strategy based on the specific nature of a question—choosing between focused, high-coverage (recall), or contrastive retrieval. By pairing this flexible, intent-driven retrieval with a structured temporal ledger to track verification and conflicts, VESTA significantly improves long-video understanding across multiple benchmarks without sacrificing agent autonomy.

What Problem Does This Paper Address?

As video lengths grow from minutes to hours, Multimodal Large Language Models (MLLMs) face a massive computational bottleneck. Processing every frame at high resolution is prohibitively expensive, yet critical evidence is often sparsely distributed or recurring across distant intervals.

To handle this, current video agents use external memories or tools to search videos. However, they suffer from a structural mismatch: they apply a uniform search behavior to every question. They treat a query looking for a single specific moment (concentrated evidence) the same as a query tracking a recurring action across an hour-long video (broad coverage), or one trying to weigh competing multiple-choice hypotheses. While some systems use rigid, step-by-step pipelines to fix this, doing so strips the agent of its ability to autonomously explore, self-correct, or adapt to intermediate findings.

Methods

The authors introduce VESTA (Video Evidence Steering and Trust Accounting), which organizes long-video analysis into a route-conditioned acquire–verify–consolidate loop.

Figure 1: Overview of VESTA’s route-conditioned acquire–verify–consolidate loop. The intent router selects focused, recall, or contrastive evidence acquisition over a shared visual–speech scene index. Retrieved scenes enter the Reasoner as provisional ref- erences, while shared evidence operations support autonomous re-retrieval and multimodal verification. The temporal evidence ledger consolidates observations into explicit evidence, coverage, and relation/gap views that guide subsequent exploration and inform answer finalization.

Key components include:

  1. Intent Routing: Before looking at the video, an LLM examines the question and answer choices to infer an evidence-acquisition policy ($\pi$). It selects one of three modes:
  2. Focused: Prioritizes precision when evidence is likely concentrated in a single temporal region.
  3. Recall: Prioritizes broad coverage to capture recurring events across the timeline.
  4. Contrastive: Generates separate queries for each multiple-choice hypothesis to ensure balanced, unbiased evidence collection.
  5. Policy-Steered Retrieval: The chosen policy queries a pre-computed, query-independent Shared Scene Index (which indexes visual frames and speech transcripts locally). Crucially, the retrieved scenes are treated only as provisional references.
  6. Agentic Exploration & Verification: A Reasoner LLM uses multimodal tools (visual inspection, speech transcription, local clip analysis) to verify these provisional references, remaining free to re-query or explore outside the initial retrieved set.
  7. Temporal Evidence Ledger: Instead of just accumulating raw tool outputs, VESTA normalizes observations into a structured ledger. The ledger tracks temporal coverage, highlights conflicts, and notes which hypotheses are supported or refuted. This compressed snapshot acts as a roadmap, exposing remaining "evidence gaps" to guide the agent's next steps.

Datasets

VESTA was evaluated on four long-video benchmarks with complementary temporal scales:

  • Video-MME-v2: 800 videos with 3,200 multi-choice questions evaluating multi-level capabilities, consistency, and coherence.
  • LongVideoBench (15–60 minute subset): Focuses heavily on localized referring reasoning within long, interleaved multimodal inputs.
  • LVBench: Features videos averaging 68 minutes, requiring agents to track entities and gather highly dispersed evidence.
  • EgoSchema-500: 3-minute egocentric video clips used to verify that VESTA’s long-range routing mechanisms do not degrade performance on short-form tasks.

Key Insights and Ideas

  • Retrieval Structure Dictates Success: The bottleneck in long-video understanding often occurs before substantive reasoning begins. If the initial retrieval phase collapses onto a single timestamp or biases toward one hypothesis, even the strongest LLM will fail.
  • Provisional vs. Verified Evidence: Existing agents often treat retrieved video segments as absolute facts. VESTA treats them as unverified candidates that the agent must explicitly cross-examine using localized visual and speech tools.
  • Ledgers Over Raw Memory: Simply appending tool outputs to an LLM's context window forces the model to repeatedly reconstruct timelines. VESTA's ledger explicitly surfaces conflicts and coverage gaps, effectively telling the agent exactly what it still needs to find.
  • Guidance Pays for Itself: By starting with a smart, question-conditioned retrieval strategy, the agent terminates its search in fewer iterations, avoiding the token-heavy cost of blindly scanning overlapping video intervals.

Key Results

VESTA consistently outperformed both standard MLLMs and state-of-the-art agentic baselines, with performance gains scaling alongside video duration:

  • Video-MME-v2: VESTA achieved an average accuracy of 50.3%, outperforming the direct agentic baseline, VideoARM (47.6%). It showed notable improvements in categories requiring temporal alignment and ordering, such as Frame-Only (+7.0) and Physical World Reasoning (+5.7).
Figure 2: Question-type analysis on Video-MME-v2. (a) Distribution of the fixed 400-question subset used for all mechanism ablations. (b) Accuracy on all 3,200 questions grouped by the benchmark’s second_head annotation. Within each pair, the upper solid bar is VESTA and the lower outlined bar is VideoARM; signed labels report VESTA mi- nus VideoARM in percentage points.
  • Long-Video Benchmarks: On the 15–60 minute LongVideoBench subset, VESTA improved performance by 6.9 points over VideoARM (72.0% vs 65.1%). On the hour-long LVBench, it achieved 68.7% (+1.5 points).
  • Short-Video Safety: On the 3-minute EgoSchema clips, VESTA matched the baseline at 73.8%, proving that its intent routing is "harm-free" when long-range search isn't necessary.
  • Ablation Studies: Removing the Temporal Evidence Ledger caused a 3.5-point drop in accuracy, while removing the Intent Router caused a 6.5-point drop, validating the importance of both components.

Limitations and Future Work

While highly effective, VESTA currently relies on a fixed, pre-defined set of three retrieval policies (focused, recall, contrastive). The authors note that expanding this policy space could allow for even finer-grained search strategies. Additionally, the current implementation relies on proprietary commercial models (OpenAI o3 and GPT-4o) for the query-time reasoning and visual verification steps; evaluating and optimizing VESTA using fully open-source model configurations remains an important direction for future research.

Summary updated

Abstract

Existing long-video agents acquire evidence through one uniform behavior, ignoring whether the required evidence is concentrated, requires broad occurrence coverage, or must discriminate competing hypotheses---which can cause failure before substantive reasoning begins. Prescribing a fine-grained solution procedure for every question is not a satisfactory remedy, as it restricts autonomous exploration. We propose VESTA, a training-free long-video agent organized as a route-conditioned acquire--verify--consolidate loop. Before exploration, an intent router infers an evidence-acquisition policy---focused, recall, or contrastive retrieval over a shared visual--speech scene index---together with an evidence-accounting policy that configures the evidence view maintained during exploration. Policy-steered retrieval yields provisional references that multimodal evidence operations convert into observations, while the Reasoner remains free to verify them, re-query using intermediate findings, or inspect regions outside the retrieved set. A temporal evidence ledger consolidates observations into an adaptive, compressed view of temporal location, provenance, coverage, conflicts, verification outcomes, and hypothesis support, exposing missing and unresolved evidence to guide subsequent acquisition; finalization prioritizes verified observations. On Video-MME-v2, VESTA improves average accuracy by 2.7 points over VideoARM and gains across all six reported metrics. On LongVideoBench, EgoSchema, and LVBench under shared query-time models, it improves by 6.9 points on the LongVideoBench long subset and 1.5 on LVBench, and matches VideoARM on EgoSchema.
BibTeX
@misc{zhang2026from,
  author = {Can Zhang and Baofeng Zhang and Xiaotian Han and Junyuan Shang and Yuchen Ding and Shuohuan Wang and Dianhai Yu and Ruirui Li},
  title = {{From Intent to Evidence: Policy-Steered Multi-Strategy Retrieval for Long-Video Agents}},
  year = {2026},
  eprint = {2608.31005},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.31005},
}
Figure