TL;DR
SAGAQA is a novel multi-video, multi-hop, and multi-modal benchmark designed to evaluate long-form narrative understanding across roughly 20 hours of continuous TV series episodes. By testing various agentic video planning strategies on this dataset, the researchers demonstrate that a new "Hybrid Planner"—which combines parallel exploration with sequential refinement—significantly outperforms traditional planning methods in grounding complex, long-range narrative questions.
What Problem Does This Paper Address?
While Multimodal Large Language Models (MLLMs) excel at short-term visual perception (e.g., describing a 5-minute clip), they struggle with long-form narrative comprehension. Existing video benchmarks typically focus on short, single-file videos or rely heavily on adjacent frames and low-level visual matching.
However, understanding complex human narratives—like those in a TV drama—requires connecting distant events, tracking long-term character arcs, and understanding causal dependencies spread across hours of content. To bridge this gap, the authors address the challenge of "big-hop" reasoning: the ability of an AI agent to logically link disparate events separated by hours of video and multiple episodes to answer high-level narrative questions.
Methods
Because processing 20 hours of video end-to-end is computationally impossible for current models, researchers rely on agentic video planners paired with Retrieval-Augmented Generation (VideoRAG). The paper formalizes and compares three distinct video planning paradigms:
- Parallel Planner: Splits a complex question into a set of independent subquestions and executes them all at once. It provides broad search coverage but struggles with iterative, multi-hop dependencies.
- Sequential Planner: Generates and executes subquestions one at a time, building on previous answers. While good at refinement, it is highly prone to "reasoning drift" and repetitive loops.
- Hybrid Planner (Proposed): Combines the strengths of both. Like a sequential planner, it dynamically selects actions step-by-step, but like a parallel planner, it generates and executes multiple independent queries simultaneously at each step. This design keeps reasoning chains short, reduces noise accumulation, and maintains the flexibility needed for multi-hop dependencies.
Datasets
The authors introduce SAGAQA, a benchmark consisting of 200 abstractive question-answering pairs spanning four years (2007–2010) of the TV show As the World Turns.
- Context Window: Each question targets a window of 20 consecutive episodes (~20 hours of video).
- Curation Pipeline: The dataset was generated synthetically using LLMs (Gemini 2.5 Flash) fed with multimodal episode summaries from the SummScreen3D dataset, followed by strict algorithmic filtering and human verification.
- Constraints: Every QA pair is strictly verified to be multi-episode (cannot be answered by one episode), multi-hop (requires connecting logical steps), and multi-modal (requires both video and dialogue).
On average, answering a single question requires four reasoning hops, with relevant events spanning a median temporal distance of 13 episodes.
Key Insights and Ideas
- The Exploration-Exploitation Trade-Off: Video agents face a fundamental trade-off. Sequential agents over-exploit early high-confidence paths and get stuck in loops, while parallel agents explore broadly but lack the depth to resolve causal chains. Hybrid planning successfully balances this trade-off.
- Visuals are Essential for Narrative: Replacing VideoRAG with TextRAG (searching only text transcripts) caused a massive drop in episode grounding metrics. This proves that dialogue alone is insufficient; visual cues (e.g., a character's facial expression, a hidden object) are vital to unlocking the full narrative.
- The Answer Synthesis Bottleneck: Interestingly, while advanced planners drastically improved episode grounding (finding the right episodes), text generation metrics like ROUGE and METEOR remained largely unchanged. This reveals that the primary bottleneck for long-form video AI is no longer just retrieval, but downstream information synthesis—the ability to aggregate information from isolated video segments into a coherent final answer.
Key Results
The planners were evaluated using Qwen3-30B-Instruct and Mistral-Small-3.1-24B. The primary metric was the F1 score for Episode Grounding (correctly identifying which episodes contained the answers):
- Hybrid Planner + VideoRAG achieved the highest performance with an F1 score of 41.02% (Qwen3) and 41.69% (Mistral).
- Parallel Planners peaked at a 34.26% F1 score (Qwen3).
- Sequential Planners lagged behind at 32.68% F1 score (Qwen3), frequently suffering from repetitive query loops.
- TextRAG (Text-only baseline) performed poorly, yielding only a 14.92% F1 score, highlighting the necessity of multimodal video data.
Limitations and Future Work
- Genre Specificity: SAGAQA is built entirely on soap operas, which feature highly specific, dialogue-driven narrative structures. These models might not generalize well to other formats like documentaries or instructional videos.
- Temporal Scale: While 20 hours is a massive step forward, real-world TV series span hundreds of episodes over multiple years. SAGAQA does not yet test "ultra-long" memory across an entire show's lifecycle.
- Granularity: Grounding is currently evaluated at the episode level. Future iterations would benefit from clip-level or frame-level annotations for more precise evaluation.
Summary updated