SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series

Galann Pennec, Zhengyuan Liu, Nicholas Asher, Philippe Muller, Nancy F. ChenarXiv 2026 · arXiv

TL;DR

SAGAQA is a novel multi-video, multi-hop, and multi-modal benchmark designed to evaluate long-form narrative understanding across roughly 20 hours of continuous TV series episodes. By testing various agentic video planning strategies on this dataset, the researchers demonstrate that a new "Hybrid Planner"—which combines parallel exploration with sequential refinement—significantly outperforms traditional planning methods in grounding complex, long-range narrative questions.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at short-term visual perception (e.g., describing a 5-minute clip), they struggle with long-form narrative comprehension. Existing video benchmarks typically focus on short, single-file videos or rely heavily on adjacent frames and low-level visual matching.

However, understanding complex human narratives—like those in a TV drama—requires connecting distant events, tracking long-term character arcs, and understanding causal dependencies spread across hours of content. To bridge this gap, the authors address the challenge of "big-hop" reasoning: the ability of an AI agent to logically link disparate events separated by hours of video and multiple episodes to answer high-level narrative questions.

Methods

Because processing 20 hours of video end-to-end is computationally impossible for current models, researchers rely on agentic video planners paired with Retrieval-Augmented Generation (VideoRAG). The paper formalizes and compares three distinct video planning paradigms:

  1. Parallel Planner: Splits a complex question into a set of independent subquestions and executes them all at once. It provides broad search coverage but struggles with iterative, multi-hop dependencies.
  2. Sequential Planner: Generates and executes subquestions one at a time, building on previous answers. While good at refinement, it is highly prone to "reasoning drift" and repetitive loops.
  3. Hybrid Planner (Proposed): Combines the strengths of both. Like a sequential planner, it dynamically selects actions step-by-step, but like a parallel planner, it generates and executes multiple independent queries simultaneously at each step. This design keeps reasoning chains short, reduces noise accumulation, and maintains the flexibility needed for multi-hop dependencies.
Existing strategies for video planning

Datasets

The authors introduce SAGAQA, a benchmark consisting of 200 abstractive question-answering pairs spanning four years (2007–2010) of the TV show As the World Turns.

  • Context Window: Each question targets a window of 20 consecutive episodes (~20 hours of video).
  • Curation Pipeline: The dataset was generated synthetically using LLMs (Gemini 2.5 Flash) fed with multimodal episode summaries from the SummScreen3D dataset, followed by strict algorithmic filtering and human verification.
  • Constraints: Every QA pair is strictly verified to be multi-episode (cannot be answered by one episode), multi-hop (requires connecting logical steps), and multi-modal (requires both video and dialogue).
Figure 1: Our data generation pipeline for SAGAQA. 1) Given a list of 20 consecutive episodes E20 and their corresponding annotated summaries from SummScreen3D, we prompt an LLM to generate synthetic QA pairs that satisfy three key criteria: multi-episode, multi-hop and multi-modal. In the example shown, the generated question requires four reasoning hops, each being logically connected to the previous one in the reasoning chain. The hops are annotated in BLUE while the part of the answer that refers to both modalities (video and dialogue) is highlighted in GREEN.

On average, answering a single question requires four reasoning hops, with relevant events spanning a median temporal distance of 13 episodes.

Statistics of SAGAQA showing episode distances and counts

Key Insights and Ideas

  • The Exploration-Exploitation Trade-Off: Video agents face a fundamental trade-off. Sequential agents over-exploit early high-confidence paths and get stuck in loops, while parallel agents explore broadly but lack the depth to resolve causal chains. Hybrid planning successfully balances this trade-off.
  • Visuals are Essential for Narrative: Replacing VideoRAG with TextRAG (searching only text transcripts) caused a massive drop in episode grounding metrics. This proves that dialogue alone is insufficient; visual cues (e.g., a character's facial expression, a hidden object) are vital to unlocking the full narrative.
  • The Answer Synthesis Bottleneck: Interestingly, while advanced planners drastically improved episode grounding (finding the right episodes), text generation metrics like ROUGE and METEOR remained largely unchanged. This reveals that the primary bottleneck for long-form video AI is no longer just retrieval, but downstream information synthesis—the ability to aggregate information from isolated video segments into a coherent final answer.

Key Results

The planners were evaluated using Qwen3-30B-Instruct and Mistral-Small-3.1-24B. The primary metric was the F1 score for Episode Grounding (correctly identifying which episodes contained the answers):

  • Hybrid Planner + VideoRAG achieved the highest performance with an F1 score of 41.02% (Qwen3) and 41.69% (Mistral).
  • Parallel Planners peaked at a 34.26% F1 score (Qwen3).
  • Sequential Planners lagged behind at 32.68% F1 score (Qwen3), frequently suffering from repetitive query loops.
  • TextRAG (Text-only baseline) performed poorly, yielding only a 14.92% F1 score, highlighting the necessity of multimodal video data.

Limitations and Future Work

  • Genre Specificity: SAGAQA is built entirely on soap operas, which feature highly specific, dialogue-driven narrative structures. These models might not generalize well to other formats like documentaries or instructional videos.
  • Temporal Scale: While 20 hours is a massive step forward, real-world TV series span hundreds of episodes over multiple years. SAGAQA does not yet test "ultra-long" memory across an entire show's lifecycle.
  • Granularity: Grounding is currently evaluated at the episode level. Future iterations would benefit from clip-level or frame-level annotations for more precise evaluation.

Summary updated

Abstract

We introduce SagaQA, a long-form video benchmark for multi-hop reasoning over full-length TV series. Existing video reasoning benchmarks often emphasize local understanding of adjacent frames or clips. SagaQA addresses this gap by requiring high-level comprehension of extended multimodal narratives in entire TV shows. A distinguishing feature of SagaQA is the granularity of its reasoning steps. Our dataset necessitates long-range reasoning hops to connect information across completely different episodes. This requires models to reason over entire events and actions, demanding a deep understanding of the show's narration and progression at a multimodal level. Motivated by recent progress in agentic methods, we further study how different planning strategies handle such complex reasoning. We categorize these approaches into three classes-Parallel, Sequential, and Hybrid planners-and evaluate their ability to generate coherent and complete reasoning plans. Our results on SagaQA suggest that hybrid planners consistently produce higher-quality plans and exhibit stronger capabilities for complex, high-level narrative understanding in TV shows.
BibTeX
@misc{pennec2026sagaqa,
  author = {Galann Pennec and Zhengyuan Liu and Nicholas Asher and Philippe Muller and Nancy F. Chen},
  title = {{SagaQA: A Multi-hop Reasoning Benchmark for Long-form Narrative Understanding in TV Series}},
  year = {2026},
  eprint = {2606.03301},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.03301},
}

Loading related content...

Figure