ForeSeek: Foresight-Driven Reasoning for Long-Form Video Understanding

Yuchen Sun, Siyuan Cheng, Shi-Zhe Chen, Yixin Qin, Jinwen Luo, Zheng WeiarXiv 2026 · arXiv

TL;DR

ForeSeek is a novel video agent that shifts long-form video understanding from "reactive retrieval" to "predictive exploration." By forming visually grounded expectations about unseen video segments and verifying them through targeted sampling, it achieves state-of-the-art accuracy while processing 2.5–4.4× fewer frames than existing search-based agents.

What Problem Does This Paper Address?

Understanding long-form video is a massive challenge for Multimodal Large Language Models (MLLMs). You can't simply feed a 30-minute video into a model; the token count would explode, and the "signal" of the answer would be drowned out by thousands of redundant frames.

To solve this, researchers have built video agents—systems that iteratively "search and retrieve" clips to answer questions. However, the authors argue that current agents are inherently reactive. They probe the timeline based on local saliency (what looks interesting right now) without a global plan. This leads to three major bottlenecks:

  1. Directionless Exploration: Searching without a hypothesis is inefficient.
  2. Unexploited Negative Evidence: If an agent looks for a goal in a specific clip and doesn't find it, it usually just discards that attempt. It doesn't use that "failure" to systematically rule out that time interval.
  3. Irrecoverable Commitment: Once an agent starts down a wrong temporal path, it often lacks a mechanism to backtrack globally.
Figure 1: Prior Agents vs. ForeSeek. Prior agents (top) reac- tively probe the video timeline without explicit expectations about unseen content, whereas ForeSeek (bottom) performs foresight-guided reasoning by maintaining predictive expec- tations that continuously guide evidence collection.

Methods

ForeSeek draws inspiration from human "predictive perception." When we watch a sports highlight, we don't just react to frames; we anticipate when a goal might happen based on the current play and then look specifically for that moment.

The ForeSeek Loop

The agent maintains a Predictive State, which includes a global "belief map" of the video timeline. The process follows three main phases:

  1. Sparse Glimpse: A quick, duration-adaptive scan of the whole video to build an initial belief.
  2. Anticipate: The model generates a visually grounded hypothesis (e.g., "At 2:1, Cape Verde is pressing; I predict the fourth goal near [76s–80s].").
  3. Verify: The agent samples frames specifically in that predicted window.
  4. If the evidence matches, it answers the question.
  5. If there is a mismatch, it treats this as "negative evidence," deprioritizes that interval in its global belief map, and redirects its search elsewhere.
Overview of the ForeSeek framework showing the two-phase reasoning process.

Two-Stage Training

To make this work, the authors developed a specialized training pipeline:

  • Stage 1: Supervised Fine-Tuning (SFT): They used a "teacher" model (Gemini-3-Flash) to synthesize reasoning trajectories where the model explicitly updates its beliefs and generates forecasts.
  • Stage 2: Efficiency-aware GSPO: They applied Group Sequence Policy Optimization (a form of Reinforcement Learning). Crucially, they introduced a length-dependent decay reward. This penalizes the model for taking too many steps, forcing it to become not just accurate, but highly efficient.

Datasets

The authors utilized a mix of high-quality long-video benchmarks for training and evaluation:

  • Training: CG-Bench, LongVideoBench, VideoHolmes, and LLaVA-Video.
  • Evaluation: Five major benchmarks including VideoMME, LongVideoBench, LVBench, VideoHolmes, and MLVU.

To ensure the model wasn't just memorizing, they tested on "Out-of-Distribution" (OOD) datasets like VideoMME and MLVU, which were never seen during the training phase.

Key Insights and Ideas

  • Negative Evidence is a Signal: One of the most powerful "aha" moments in the paper is treating a failed search as informative. By explicitly ruling out time intervals, the agent narrows the search space much faster than a random or saliency-based search.
  • Falsifiability: By forcing the agent to state what it expects to see before it looks, the system becomes self-correcting. A mismatch between expectation and observation triggers an immediate pivot in strategy.
  • Efficiency vs. "Chattiness": The researchers found that SFT alone makes models smart but "chatty"—they tend to look at more frames than necessary because more data usually helps accuracy. The RL stage (GSPO) was essential to "trim the fat" and achieve high accuracy with a tiny frame budget.

Key Results

ForeSeek sets a new bar for the efficiency-accuracy trade-off in video understanding.

  • Frame Efficiency: ForeSeek-GSPO uses significantly fewer frames (22–31 frames) compared to search-based agents like EVA (55–137 frames) while maintaining superior accuracy.
  • Performance: On the VideoHolmes benchmark, ForeSeek achieved 53.2% accuracy using only 22 frames. For context, the much larger GPT-4o achieved only 42.0% accuracy on the same task using 32 frames.
  • Generalization: Even on OOD benchmarks like VideoMME, ForeSeek outperformed specialized agents like Conan and FrameThinker, proving that the "foresight" strategy is a generalizable reasoning skill, not just a dataset-specific trick.
Performance comparison across five benchmarks showing ForeSeek's superior efficiency and accuracy.

Limitations and Future Work

While ForeSeek is a major step forward, the authors note some limitations:

  • Backbone Dependency: The agent's performance is still ultimately capped by the underlying MLLM's (Qwen3-VL-8B) ability to perceive individual frames.
  • Complex Causal Chains: While great at temporal localization, extremely complex multi-step causal reasoning (e.g., "Why did the character change their mind in scene 5?") remains a challenge for all current agents.
  • Future Directions: The authors suggest that this foresight-driven approach could be extended to real-time streaming video or integrated with more complex tool-use (like OCR or action recognition) to further enhance its "detective" capabilities.

Summary updated

Abstract

Long-form video understanding, as a fundamental yet crucial vision task, is being revolutionized by multimodal large language model (MLLM)-based video agents. However, the retrieval-driven nature inherent to these agents creates a significant bottleneck, forcing them to reactively fetch clips based on local saliency without a global temporal plan. To address this, we present ForeSeek, a foresight-driven video agent that transforms long-video reasoning from reactive retrieval into predictive exploration. Rather than blindly probing the timeline, ForeSeek forms visually grounded expectations over unseen temporal regions, verifies them through targeted observations, and continuously refines its search based on emerging evidence, where failed expectations serve as informative signals for principled backtracking. Furthermore, we develop a two-stage training framework that equips MLLMs with foresight-guided reasoning and efficient decision making. Extensive experiments on five long-video benchmarks demonstrate that ForeSeek consistently improves the efficiency--accuracy trade-off of long-video reasoning, achieving competitive or superior accuracy while observing $2.5$--$4.4\times$ fewer frames than the strongest search-based agent. Code is available at https://github.com/harry0021/ForeSeek
BibTeX
@misc{sun2026foreseek,
  author = {Yuchen Sun and Siyuan Cheng and Shi-Zhe Chen and Yixin Qin and Jinwen Luo and Zheng Wei},
  title = {{ForeSeek: Foresight-Driven Reasoning for Long-Form Video Understanding}},
  year = {2026},
  eprint = {2611.26024},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26024},
}
Figure