TL;DR
SVAgent is a novel multi-agent framework that mimics human video comprehension to tackle long-video question answering. By combining an evolving, query-guided storyline with independent cross-modal verification and iterative refinement, it achieves substantial performance gains (5.5%–11.5%) over state-of-the-art video large language models while using significantly fewer frames.
What Problem Does This Paper Address?
Understanding long videos is a massive challenge for current AI. Video Large Language Models (Video MLLMs) are severely constrained by context windows and token limits. To cope, existing methods usually compress videos into text captions, retrieve a sparse set of keyframes, or build rigid event graphs.
However, as shown in Figure 1, these approaches suffer from "temporal collapse" (losing track of the sequence of events), fragmented reasoning, or a lack of flexibility. Crucially, they don't reason the way humans do—by tracking a continuous, evolving storyline and cross-checking visual and textual cues to resolve ambiguities.
Methods
SVAgent introduces a collaborative multi-agent loop designed to break down long-video reasoning into four cooperative stages:
- Storyline Generation (Block 1): A Storyline Agent synthesizes sampled frames into a compact, query-aware narrative scaffold, preserving global temporal structure.
- Hypothesis Formation & DPPs (Block 2): A Hypothesis Agent proposes a tentative answer and identifies supporting evidence. This evidence, along with the user query, guides two independent Determinantal Point Processes (DPPs) to select keyframe subsets. The intersection ratio of these subsets determines if the hypothesis is plausible enough to proceed.
- Cross-Modal Decision-Making (Block 3): To prevent "modality drift" (where text and vision predictions diverge), a Text Decision Agent and a Visual Decision Agent independently evaluate the evidence. A Meta-Decision Agent then acts as a referee, checking for cross-modal consistency and resolving conflicts.
- Suggestion Agent (Block 4): If the meta-decision agent detects inconsistency or insufficient evidence, a Suggestion Agent analyzes the failure history and targets unexplored or highly uncertain video segments to refine the storyline in the next iteration.
Datasets
The framework was evaluated on four challenging long-video understanding benchmarks:
- Video-MME: An evaluation suite spanning videos from 11 seconds to 1 hour to assess real-world comprehension.
- MLVU: A benchmark focusing on multi-stage events and long-range temporal dependencies (3 minutes to 2 hours).
- LongVideoBench: A dataset targeting referred and multi-hop reasoning over long frame sequences.
- LVBench: A benchmark emphasizing long-range temporal grounding and multi-event reasoning.
Key Insights and Ideas
- Storylines over Isolated Frames: Maintaining an explicit, evolving narrative scaffold prevents the model from getting lost in repetitive or visually dominant but irrelevant scenes.
- Divide and Conquer for Modalities: Fusing text and vision too early can cause one modality to blind the other. Keeping the text and vision decision pathways separate until the final meta-decision stage significantly reduces error propagation and information leakage.
- Quality over Quantity: Long-video reasoning benefits more from smart, iterative frame selection than from simply dumping more frames into the context window.
Key Results
- State-of-the-Art Boosts: SVAgent consistently outperforms baseline models (like Qwen2.5-VL and Qwen3-VL) across all sizes (3B to 8B), yielding 5.5% to 11.5% accuracy improvements.
- Frame Efficiency: As shown in Figure 4, on Video-MME, SVAgent with just 8 frames achieves 60.7% accuracy, outperforming uniform sampling at 32 frames (59.8%).
- Ablation Validation: Removing the Storyline Agent caused the largest performance drop (4–8 points), proving that a global temporal scaffold is critical for long-range reasoning.
Limitations and Future Work
While SVAgent is highly accurate and efficient with frames, its iterative nature introduces a trade-off between accuracy and latency. Increasing the strictness of evidence verification (the intersection ratio threshold $\tau$) or the number of refinement iterations ($T$) improves accuracy but increases processing time due to repeated DPP calculations and multi-agent reasoning cycles. Future work will likely focus on optimizing agent communication overhead to reduce this computational latency.
Summary updated