TL;DR
ForeSeek is a novel video agent that shifts long-form video understanding from "reactive retrieval" to "predictive exploration." By forming visually grounded expectations about unseen video segments and verifying them through targeted sampling, it achieves state-of-the-art accuracy while processing 2.5–4.4× fewer frames than existing search-based agents.
What Problem Does This Paper Address?
Understanding long-form video is a massive challenge for Multimodal Large Language Models (MLLMs). You can't simply feed a 30-minute video into a model; the token count would explode, and the "signal" of the answer would be drowned out by thousands of redundant frames.
To solve this, researchers have built video agents—systems that iteratively "search and retrieve" clips to answer questions. However, the authors argue that current agents are inherently reactive. They probe the timeline based on local saliency (what looks interesting right now) without a global plan. This leads to three major bottlenecks:
- Directionless Exploration: Searching without a hypothesis is inefficient.
- Unexploited Negative Evidence: If an agent looks for a goal in a specific clip and doesn't find it, it usually just discards that attempt. It doesn't use that "failure" to systematically rule out that time interval.
- Irrecoverable Commitment: Once an agent starts down a wrong temporal path, it often lacks a mechanism to backtrack globally.
Methods
ForeSeek draws inspiration from human "predictive perception." When we watch a sports highlight, we don't just react to frames; we anticipate when a goal might happen based on the current play and then look specifically for that moment.
The ForeSeek Loop
The agent maintains a Predictive State, which includes a global "belief map" of the video timeline. The process follows three main phases:
- Sparse Glimpse: A quick, duration-adaptive scan of the whole video to build an initial belief.
- Anticipate: The model generates a visually grounded hypothesis (e.g., "At 2:1, Cape Verde is pressing; I predict the fourth goal near [76s–80s].").
- Verify: The agent samples frames specifically in that predicted window.
- If the evidence matches, it answers the question.
- If there is a mismatch, it treats this as "negative evidence," deprioritizes that interval in its global belief map, and redirects its search elsewhere.
Two-Stage Training
To make this work, the authors developed a specialized training pipeline:
- Stage 1: Supervised Fine-Tuning (SFT): They used a "teacher" model (Gemini-3-Flash) to synthesize reasoning trajectories where the model explicitly updates its beliefs and generates forecasts.
- Stage 2: Efficiency-aware GSPO: They applied Group Sequence Policy Optimization (a form of Reinforcement Learning). Crucially, they introduced a length-dependent decay reward. This penalizes the model for taking too many steps, forcing it to become not just accurate, but highly efficient.
Datasets
The authors utilized a mix of high-quality long-video benchmarks for training and evaluation:
- Training: CG-Bench, LongVideoBench, VideoHolmes, and LLaVA-Video.
- Evaluation: Five major benchmarks including VideoMME, LongVideoBench, LVBench, VideoHolmes, and MLVU.
To ensure the model wasn't just memorizing, they tested on "Out-of-Distribution" (OOD) datasets like VideoMME and MLVU, which were never seen during the training phase.
Key Insights and Ideas
- Negative Evidence is a Signal: One of the most powerful "aha" moments in the paper is treating a failed search as informative. By explicitly ruling out time intervals, the agent narrows the search space much faster than a random or saliency-based search.
- Falsifiability: By forcing the agent to state what it expects to see before it looks, the system becomes self-correcting. A mismatch between expectation and observation triggers an immediate pivot in strategy.
- Efficiency vs. "Chattiness": The researchers found that SFT alone makes models smart but "chatty"—they tend to look at more frames than necessary because more data usually helps accuracy. The RL stage (GSPO) was essential to "trim the fat" and achieve high accuracy with a tiny frame budget.
Key Results
ForeSeek sets a new bar for the efficiency-accuracy trade-off in video understanding.
- Frame Efficiency: ForeSeek-GSPO uses significantly fewer frames (22–31 frames) compared to search-based agents like EVA (55–137 frames) while maintaining superior accuracy.
- Performance: On the VideoHolmes benchmark, ForeSeek achieved 53.2% accuracy using only 22 frames. For context, the much larger GPT-4o achieved only 42.0% accuracy on the same task using 32 frames.
- Generalization: Even on OOD benchmarks like VideoMME, ForeSeek outperformed specialized agents like Conan and FrameThinker, proving that the "foresight" strategy is a generalizable reasoning skill, not just a dataset-specific trick.
Limitations and Future Work
While ForeSeek is a major step forward, the authors note some limitations:
- Backbone Dependency: The agent's performance is still ultimately capped by the underlying MLLM's (Qwen3-VL-8B) ability to perceive individual frames.
- Complex Causal Chains: While great at temporal localization, extremely complex multi-step causal reasoning (e.g., "Why did the character change their mind in scene 5?") remains a challenge for all current agents.
- Future Directions: The authors suggest that this foresight-driven approach could be extended to real-time streaming video or integrated with more complex tool-use (like OCR or action recognition) to further enhance its "detective" capabilities.
Summary updated