TL;DR
VESTA is a training-free long-video AI agent that dynamically adapts its evidence-gathering strategy based on the specific nature of a question—choosing between focused, high-coverage (recall), or contrastive retrieval. By pairing this flexible, intent-driven retrieval with a structured temporal ledger to track verification and conflicts, VESTA significantly improves long-video understanding across multiple benchmarks without sacrificing agent autonomy.
What Problem Does This Paper Address?
As video lengths grow from minutes to hours, Multimodal Large Language Models (MLLMs) face a massive computational bottleneck. Processing every frame at high resolution is prohibitively expensive, yet critical evidence is often sparsely distributed or recurring across distant intervals.
To handle this, current video agents use external memories or tools to search videos. However, they suffer from a structural mismatch: they apply a uniform search behavior to every question. They treat a query looking for a single specific moment (concentrated evidence) the same as a query tracking a recurring action across an hour-long video (broad coverage), or one trying to weigh competing multiple-choice hypotheses. While some systems use rigid, step-by-step pipelines to fix this, doing so strips the agent of its ability to autonomously explore, self-correct, or adapt to intermediate findings.
Methods
The authors introduce VESTA (Video Evidence Steering and Trust Accounting), which organizes long-video analysis into a route-conditioned acquire–verify–consolidate loop.
Key components include:
- Intent Routing: Before looking at the video, an LLM examines the question and answer choices to infer an evidence-acquisition policy ($\pi$). It selects one of three modes:
- Focused: Prioritizes precision when evidence is likely concentrated in a single temporal region.
- Recall: Prioritizes broad coverage to capture recurring events across the timeline.
- Contrastive: Generates separate queries for each multiple-choice hypothesis to ensure balanced, unbiased evidence collection.
- Policy-Steered Retrieval: The chosen policy queries a pre-computed, query-independent Shared Scene Index (which indexes visual frames and speech transcripts locally). Crucially, the retrieved scenes are treated only as provisional references.
- Agentic Exploration & Verification: A Reasoner LLM uses multimodal tools (visual inspection, speech transcription, local clip analysis) to verify these provisional references, remaining free to re-query or explore outside the initial retrieved set.
- Temporal Evidence Ledger: Instead of just accumulating raw tool outputs, VESTA normalizes observations into a structured ledger. The ledger tracks temporal coverage, highlights conflicts, and notes which hypotheses are supported or refuted. This compressed snapshot acts as a roadmap, exposing remaining "evidence gaps" to guide the agent's next steps.
Datasets
VESTA was evaluated on four long-video benchmarks with complementary temporal scales:
- Video-MME-v2: 800 videos with 3,200 multi-choice questions evaluating multi-level capabilities, consistency, and coherence.
- LongVideoBench (15–60 minute subset): Focuses heavily on localized referring reasoning within long, interleaved multimodal inputs.
- LVBench: Features videos averaging 68 minutes, requiring agents to track entities and gather highly dispersed evidence.
- EgoSchema-500: 3-minute egocentric video clips used to verify that VESTA’s long-range routing mechanisms do not degrade performance on short-form tasks.
Key Insights and Ideas
- Retrieval Structure Dictates Success: The bottleneck in long-video understanding often occurs before substantive reasoning begins. If the initial retrieval phase collapses onto a single timestamp or biases toward one hypothesis, even the strongest LLM will fail.
- Provisional vs. Verified Evidence: Existing agents often treat retrieved video segments as absolute facts. VESTA treats them as unverified candidates that the agent must explicitly cross-examine using localized visual and speech tools.
- Ledgers Over Raw Memory: Simply appending tool outputs to an LLM's context window forces the model to repeatedly reconstruct timelines. VESTA's ledger explicitly surfaces conflicts and coverage gaps, effectively telling the agent exactly what it still needs to find.
- Guidance Pays for Itself: By starting with a smart, question-conditioned retrieval strategy, the agent terminates its search in fewer iterations, avoiding the token-heavy cost of blindly scanning overlapping video intervals.
Key Results
VESTA consistently outperformed both standard MLLMs and state-of-the-art agentic baselines, with performance gains scaling alongside video duration:
- Video-MME-v2: VESTA achieved an average accuracy of 50.3%, outperforming the direct agentic baseline, VideoARM (47.6%). It showed notable improvements in categories requiring temporal alignment and ordering, such as Frame-Only (+7.0) and Physical World Reasoning (+5.7).
- Long-Video Benchmarks: On the 15–60 minute LongVideoBench subset, VESTA improved performance by 6.9 points over VideoARM (72.0% vs 65.1%). On the hour-long LVBench, it achieved 68.7% (+1.5 points).
- Short-Video Safety: On the 3-minute EgoSchema clips, VESTA matched the baseline at 73.8%, proving that its intent routing is "harm-free" when long-range search isn't necessary.
- Ablation Studies: Removing the Temporal Evidence Ledger caused a 3.5-point drop in accuracy, while removing the Intent Router caused a 6.5-point drop, validating the importance of both components.
Limitations and Future Work
While highly effective, VESTA currently relies on a fixed, pre-defined set of three retrieval policies (focused, recall, contrastive). The authors note that expanding this policy space could allow for even finer-grained search strategies. Additionally, the current implementation relies on proprietary commercial models (OpenAI o3 and GPT-4o) for the query-time reasoning and visual verification steps; evaluating and optimizing VESTA using fully open-source model configurations remains an important direction for future research.
Summary updated