SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration

Zhongyu Yang, Zuhao Yang, Shuo Zhan, Tan Yue, Wei Pang, Yingfang Yuan

TL;DR

SVAgent is a novel multi-agent framework that mimics human video comprehension to tackle long-video question answering. By combining an evolving, query-guided storyline with independent cross-modal verification and iterative refinement, it achieves substantial performance gains (5.5%–11.5%) over state-of-the-art video large language models while using significantly fewer frames.

What Problem Does This Paper Address?

Understanding long videos is a massive challenge for current AI. Video Large Language Models (Video MLLMs) are severely constrained by context windows and token limits. To cope, existing methods usually compress videos into text captions, retrieve a sparse set of keyframes, or build rigid event graphs.

However, as shown in Figure 1, these approaches suffer from "temporal collapse" (losing track of the sequence of events), fragmented reasoning, or a lack of flexibility. Crucially, they don't reason the way humans do—by tracking a continuous, evolving storyline and cross-checking visual and textual cues to resolve ambiguities.

Figure 1. Comparison of existing video reasoning paradigms and our proposed framework. (a) Caption-based methods rely on text summaries, which often result in temporal collapse and insufficient visual grounding. (b) Keyframe retrieval methods focus on top-k evidence, often leading to fragmented reasoning. (c) Event/graph-based approaches depend on fixed structures, which limits adaptability. (d) We propose SVAgent, which unifies storyline reconstruction and cross-modal decision-making through multi-agent collaboration, recovers temporal narratives, ensures causal completeness via DPPs, and enables adaptive self-reflective reasoning beyond static paradigms.

Methods

SVAgent introduces a collaborative multi-agent loop designed to break down long-video reasoning into four cooperative stages:

  1. Storyline Generation (Block 1): A Storyline Agent synthesizes sampled frames into a compact, query-aware narrative scaffold, preserving global temporal structure.
  2. Hypothesis Formation & DPPs (Block 2): A Hypothesis Agent proposes a tentative answer and identifies supporting evidence. This evidence, along with the user query, guides two independent Determinantal Point Processes (DPPs) to select keyframe subsets. The intersection ratio of these subsets determines if the hypothesis is plausible enough to proceed.
  3. Cross-Modal Decision-Making (Block 3): To prevent "modality drift" (where text and vision predictions diverge), a Text Decision Agent and a Visual Decision Agent independently evaluate the evidence. A Meta-Decision Agent then acts as a referee, checking for cross-modal consistency and resolving conflicts.
  4. Suggestion Agent (Block 4): If the meta-decision agent detects inconsistency or insufficient evidence, a Suggestion Agent analyzes the failure history and targets unexplored or highly uncertain video segments to refine the storyline in the next iteration.
SVAgent Framework Overview

Datasets

The framework was evaluated on four challenging long-video understanding benchmarks:

  • Video-MME: An evaluation suite spanning videos from 11 seconds to 1 hour to assess real-world comprehension.
  • MLVU: A benchmark focusing on multi-stage events and long-range temporal dependencies (3 minutes to 2 hours).
  • LongVideoBench: A dataset targeting referred and multi-hop reasoning over long frame sequences.
  • LVBench: A benchmark emphasizing long-range temporal grounding and multi-event reasoning.

Key Insights and Ideas

  • Storylines over Isolated Frames: Maintaining an explicit, evolving narrative scaffold prevents the model from getting lost in repetitive or visually dominant but irrelevant scenes.
  • Divide and Conquer for Modalities: Fusing text and vision too early can cause one modality to blind the other. Keeping the text and vision decision pathways separate until the final meta-decision stage significantly reduces error propagation and information leakage.
  • Quality over Quantity: Long-video reasoning benefits more from smart, iterative frame selection than from simply dumping more frames into the context window.

Key Results

  • State-of-the-Art Boosts: SVAgent consistently outperforms baseline models (like Qwen2.5-VL and Qwen3-VL) across all sizes (3B to 8B), yielding 5.5% to 11.5% accuracy improvements.
  • Frame Efficiency: As shown in Figure 4, on Video-MME, SVAgent with just 8 frames achieves 60.7% accuracy, outperforming uniform sampling at 32 frames (59.8%).
  • Ablation Validation: Removing the Storyline Agent caused the largest performance drop (4–8 points), proving that a global temporal scaffold is critical for long-range reasoning.
Figure 4. Comparison of SVAgent and uniform sampling under different frame budgets on Video-MME.

Limitations and Future Work

While SVAgent is highly accurate and efficient with frames, its iterative nature introduces a trade-off between accuracy and latency. Increasing the strictness of evidence verification (the intersection ratio threshold $\tau$) or the number of refinement iterations ($T$) improves accuracy but increases processing time due to repeated DPP calculations and multi-agent reasoning cycles. Future work will likely focus on optimizing agent communication overhead to reduce this computational latency.

Summary updated

Abstract

Video question answering (VideoQA) is a challenging task that requires integrating spatial, temporal, and semantic information to capture the complex dynamics of video sequences. Although recent advances have introduced various approaches for video understanding, most existing methods still rely on locating relevant frames to answer questions rather than reasoning through the evolving storyline as humans do. Humans naturally interpret videos through coherent storylines, an ability that is crucial for making robust and contextually grounded predictions. To address this gap, we propose SVAgent, a storyline-guided cross-modal multi-agent framework for VideoQA. The storyline agent progressively constructs a narrative representation based on frames suggested by a refinement suggestion agent that analyzes historical failures. In addition, cross-modal decision agents independently predict answers from visual and textual modalities under the guidance of the evolving storyline. Their outputs are then evaluated by a meta-agent to align cross-modal predictions and enhance reasoning robustness and answer consistency. Experimental results demonstrate that SVAgent achieves superior performance and interpretability by emulating human-like storyline reasoning in video understanding.
BibTeX
@inproceedings{yang2026svagent,
  author = {Zhongyu Yang and Zuhao Yang and Shuo Zhan and Tan Yue and Wei Pang and Yingfang Yuan},
  title = {{SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=kq3DJqwH2l},
}
Figure