VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking

Jingyang Lin, Jialian Wu, Jiang Liu, Ximeng Sun, Ze Wang, Xiaodong Yu, Jiebo Luo, Zicheng Liu, Emad Barsoum

TL;DR

VideoSeek is a long-horizon video understanding agent that rejects the computationally expensive status quo of densely parsing every video frame. By mimicking human visual search through an iterative "think-act-observe" loop and a multi-granular toolkit, VideoSeek actively hunts for answer-critical evidence. The result is state-of-the-art accuracy on long-form video benchmarks while processing up to 99% fewer frames than traditional dense-parsing methods.

What Problem Does This Paper Address?

Analyzing long-form video (e.g., hour-long movies or lectures) using Large Multimodal Models (LMMs) is notoriously expensive. Most current approaches rely on "greedy parsing"—either uniformly sampling hundreds of frames or running heavy preprocessing pipelines to convert every few seconds of video into text descriptions or structured databases.

However, the authors point out a glaring inefficiency: on popular long-video benchmarks like LVBench, over 80% of questions can be answered by looking at less than 5% of the total video. Densely processing the remaining 95% is a massive waste of compute. VideoSeek addresses this gap by introducing an agentic framework that leverages the inherent temporal and causal logic of video to selectively "seek" out only the relevant segments, just like a human would when fast-forwarding through a clip to find a specific scene.

Methods

VideoSeek operates within a ReAct-style (think-act-observe) loop powered by a central "thinking" LLM (defaulting to GPT-5). Instead of relying on a pre-built video database or a rigid memory buffer, the agent maintains a running trajectory of its thoughts, actions, and observations directly within its conversation history.

At each turn, the agent reasons about the user's query, determines what information is missing, and invokes one of three specialized tools from its multi-granular toolkit:

  1. <overview>: Uniformly samples a very sparse set of frames across the entire timeline to build a coarse storyline and map out the video's structure.
  2. <skim>: Quickly scans a broad candidate segment at low cost to verify if query-relevant evidence is nearby.
  3. <focus>: Zooms in on a short clip with high-density inspection (1 FPS) to extract fine-grained details (e.g., reading text on a sign or identifying a specific action).
Figure 2. Toolkit of the VideoSeek agent, including <overview>, <skim>, and <focus> tools. Left: <overview> rapidly scans the entire video to build a coarse storyline and highlight promising intervals. Middle: <skim> takes a quick glance at these candidate intervals (i.e., t1 to t2) at low cost to check whether query-relevant evidence is nearby. Right: <focus> zooms in on a fine-grained clip (i.e., t3 to t4) with dense inspection to obtain answer-critical observations. Red, blue, and green boxes denote frames viewed by <overview>, <skim>, and <focus>, respectively, while gray boxes indicate unseen frames.

The loop repeats adaptively until the agent gathers sufficient evidence to confidently output the final answer or hits a maximum turn limit.

Datasets

The authors evaluated VideoSeek across four challenging long-form video understanding and complex reasoning benchmarks:

  • LVBench: Focuses on long-term memory and extended comprehension across hour-long videos.
  • Video-MME (Long Subset): A comprehensive multimodal benchmark evaluating various video types over long temporal ranges.
  • LongVideoBench: Evaluates detailed retrieval and reasoning over web videos up to an hour long.
  • Video-Holmes: A complex video reasoning benchmark consisting of manually annotated suspense short films that require models to actively locate and connect scattered visual clues.

Key Insights and Ideas

  • Subtitles as Logic Maps: When subtitles are available, VideoSeek's frame usage drops dramatically while its accuracy shoots up. Subtitles provide an explicit textual storyline, allowing the agent to reason about the video's logic flow and jump straight to answer-critical segments without scanning redundant visual content.
  • The Criticality of Reasoning Quality: The choice of the underlying "thinking" LLM is paramount. Ablation studies showed that replacing a strong model (GPT-5) with a weaker one (like GPT-4.1) caused the agent to become overconfident, stopping its search prematurely with insufficient evidence. Conversely, replacing it with o4-mini led to more tool calls and frame processing, but failed to translate that extra data into better judgment.
  • Coarse-to-Fine Synergy: Ablation of the toolkit revealed that the <overview> tool is the most vital component. Without a global pass to capture the initial logic flow, the agent's ability to localize subsequent search areas degrades severely.

Key Results

VideoSeek consistently outperformed both standalone LMMs and dense video agentic models while operating on a fraction of the frame budget:

  • LVBench (with subtitles): VideoSeek achieved a state-of-the-art 76.7% accuracy using an average of just 27.2 frames. For comparison, the top-performing peer agent, DVD, achieved 76.0% but required 8,074 frames—meaning VideoSeek matched or exceeded its performance using roughly 0.3% of the visual budget.
  • Video-MME: Without subtitles, VideoSeek scored 70.1% using only 60.9 frames, outperforming Gemini 1.5 Pro (67.4% using 1,233 frames). With subtitles, its frame count dropped to a mere 15.9 frames while accuracy rose to 81.2%.
  • Efficiency Gains: Across all long-form benchmarks, VideoSeek improved upon its base model (GPT-5 with uniform sampling) by up to 10.2 absolute percentage points while slashing frame usage by 76% to 96%.

Limitations and Future Work

The primary limitation of VideoSeek is its reliance on logic-driven navigation. The authors note that the model is less suited for tasks involving unexpected or highly localized surprising moments (such as anomaly detection in security footage). If an event does not follow a predictable narrative or causal flow, the agent cannot easily anticipate where to look based on a coarse overview.

Future work will focus on expanding the agent's capabilities to better handle these rare, sudden, and unexpected visual events without sacrificing its signature frame efficiency.

Summary updated

Abstract

Video agentic models have advanced challenging video-language tasks. However, most agentic approaches still heavily rely on greedy parsing over densely sampled video frames, resulting in high computational cost. We present VideoSeek, a long-horizon video agent that leverages video logic flow to actively seek answer-critical evidence instead of exhaustively parsing the full video. This insight allows the model to use far fewer frames while maintaining, or even improving, its video understanding capability. VideoSeek operates in a think-act-observe loop with a well-designed toolkit for collecting multi-granular video observations. This design enables query-aware exploration over accumulated observations and supports practical video understanding and reasoning. Experiments on four challenging video understanding and reasoning benchmarks demonstrate that VideoSeek achieves strong accuracy while using far fewer frames than prior video agents and standalone LMMs. Notably, VideoSeek achieves a 10.2 absolute points improvement on LVBench over its base model, GPT-5, while using 93% fewer frames. Further analysis highlights the significance of leveraging video logic flow, strong reasoning capability, and the complementary roles of toolkit design.
BibTeX
@inproceedings{lin2026videoseek,
  author = {Jingyang Lin and Jialian Wu and Jiang Liu and Ximeng Sun and Ze Wang and Xiaodong Yu and Jiebo Luo and Zicheng Liu and Emad Barsoum},
  title = {{VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=EjrCNF8Xsr},
}
Figure