TL;DR
VideoSeek is a long-horizon video understanding agent that rejects the computationally expensive status quo of densely parsing every video frame. By mimicking human visual search through an iterative "think-act-observe" loop and a multi-granular toolkit, VideoSeek actively hunts for answer-critical evidence. The result is state-of-the-art accuracy on long-form video benchmarks while processing up to 99% fewer frames than traditional dense-parsing methods.
What Problem Does This Paper Address?
Analyzing long-form video (e.g., hour-long movies or lectures) using Large Multimodal Models (LMMs) is notoriously expensive. Most current approaches rely on "greedy parsing"—either uniformly sampling hundreds of frames or running heavy preprocessing pipelines to convert every few seconds of video into text descriptions or structured databases.
However, the authors point out a glaring inefficiency: on popular long-video benchmarks like LVBench, over 80% of questions can be answered by looking at less than 5% of the total video. Densely processing the remaining 95% is a massive waste of compute. VideoSeek addresses this gap by introducing an agentic framework that leverages the inherent temporal and causal logic of video to selectively "seek" out only the relevant segments, just like a human would when fast-forwarding through a clip to find a specific scene.
Methods
VideoSeek operates within a ReAct-style (think-act-observe) loop powered by a central "thinking" LLM (defaulting to GPT-5). Instead of relying on a pre-built video database or a rigid memory buffer, the agent maintains a running trajectory of its thoughts, actions, and observations directly within its conversation history.
At each turn, the agent reasons about the user's query, determines what information is missing, and invokes one of three specialized tools from its multi-granular toolkit:
<overview>: Uniformly samples a very sparse set of frames across the entire timeline to build a coarse storyline and map out the video's structure.<skim>: Quickly scans a broad candidate segment at low cost to verify if query-relevant evidence is nearby.<focus>: Zooms in on a short clip with high-density inspection (1 FPS) to extract fine-grained details (e.g., reading text on a sign or identifying a specific action).
The loop repeats adaptively until the agent gathers sufficient evidence to confidently output the final answer or hits a maximum turn limit.
Datasets
The authors evaluated VideoSeek across four challenging long-form video understanding and complex reasoning benchmarks:
- LVBench: Focuses on long-term memory and extended comprehension across hour-long videos.
- Video-MME (Long Subset): A comprehensive multimodal benchmark evaluating various video types over long temporal ranges.
- LongVideoBench: Evaluates detailed retrieval and reasoning over web videos up to an hour long.
- Video-Holmes: A complex video reasoning benchmark consisting of manually annotated suspense short films that require models to actively locate and connect scattered visual clues.
Key Insights and Ideas
- Subtitles as Logic Maps: When subtitles are available, VideoSeek's frame usage drops dramatically while its accuracy shoots up. Subtitles provide an explicit textual storyline, allowing the agent to reason about the video's logic flow and jump straight to answer-critical segments without scanning redundant visual content.
- The Criticality of Reasoning Quality: The choice of the underlying "thinking" LLM is paramount. Ablation studies showed that replacing a strong model (GPT-5) with a weaker one (like GPT-4.1) caused the agent to become overconfident, stopping its search prematurely with insufficient evidence. Conversely, replacing it with o4-mini led to more tool calls and frame processing, but failed to translate that extra data into better judgment.
- Coarse-to-Fine Synergy: Ablation of the toolkit revealed that the
<overview>tool is the most vital component. Without a global pass to capture the initial logic flow, the agent's ability to localize subsequent search areas degrades severely.
Key Results
VideoSeek consistently outperformed both standalone LMMs and dense video agentic models while operating on a fraction of the frame budget:
- LVBench (with subtitles): VideoSeek achieved a state-of-the-art 76.7% accuracy using an average of just 27.2 frames. For comparison, the top-performing peer agent, DVD, achieved 76.0% but required 8,074 frames—meaning VideoSeek matched or exceeded its performance using roughly 0.3% of the visual budget.
- Video-MME: Without subtitles, VideoSeek scored 70.1% using only 60.9 frames, outperforming Gemini 1.5 Pro (67.4% using 1,233 frames). With subtitles, its frame count dropped to a mere 15.9 frames while accuracy rose to 81.2%.
- Efficiency Gains: Across all long-form benchmarks, VideoSeek improved upon its base model (GPT-5 with uniform sampling) by up to 10.2 absolute percentage points while slashing frame usage by 76% to 96%.
Limitations and Future Work
The primary limitation of VideoSeek is its reliance on logic-driven navigation. The authors note that the model is less suited for tasks involving unexpected or highly localized surprising moments (such as anomaly detection in security footage). If an event does not follow a predictable narrative or causal flow, the agent cannot easily anticipate where to look based on a coarse overview.
Future work will focus on expanding the agent's capabilities to better handle these rare, sudden, and unexpected visual events without sacrificing its signature frame efficiency.
Summary updated