Abstract
Multimodal Large Language Models (MLLMs) face significant challenges when processing long-form video content due to the "resource wall" that prevents simultaneous processing of all frames. This paper introduces Scene-Localized Frame Grouping (SLFG), a plug-and-play framework that organizes video frames into semantically coherent "scenes" to facilitate better reasoning. Alongside SLFG, the authors present SceneQA, a new task for scene-level comprehension, and LVSQA, a benchmark dataset of videos exceeding 30 minutes. SLFG demonstrates substantial performance gains across multiple long-video benchmarks without requiring model retraining.
Introduction
Most existing Video Question Answering (VideoQA) benchmarks focus on "needle-in-a-haystack" retrieval, asking models to locate specific objects in single frames. However, true video understanding requires a "scene-level" grasp of actions and relationships over time. Current MLLMs often downsample long videos so aggressively that they lose the fine-grained details necessary for complex narrative reasoning.
The SLFG Framework
The proposed SLFG method mimics human cognition by treating the "scene" as the primary unit of analysis. It operates through a four-stage pipeline designed to bridge the gap between raw visual data and high-level reasoning:
- Group Frames Description: The video is densely sampled and grouped, with a base MLLM generating textual descriptions for each group.
- Scene Generation: An LLM (e.g., Qwen2.5-7B) abstracts these descriptions into concise scene summaries, reducing noise and redundancy.
- Scene Localization: A text embedding model (BGE-M3) identifies the most relevant scenes by comparing summaries to the user's question.
- Dynamic Frame Grouping: The system merges consecutive high-scoring segments and expands the temporal window to ensure the model captures the full context of an event rather than just a snapshot.
Experimental Results and Benchmarks
The authors introduced the LVSQA (Long Video Scene-level Question Answering) dataset, featuring 100 videos (30+ minutes each) and 500 human-refined QA pairs focused on detail recognition and causal reasoning. In evaluations, SLFG consistently enhanced state-of-the-art models:
- LVSQA: LLaVA-Video + SLFG reached 63.4% accuracy, a significant improvement over the baseline.
- LVBench: LLaVA-OneVision achieved a 9.8% relative increase in accuracy when using the SLFG framework.
- Efficiency: The marginal cost per question decreases significantly in multi-query scenarios, as the visual preprocessing and scene generation stages are performed only once per video.
Conclusion and Future Work
SLFG offers a scalable, training-free solution for long-video understanding by leveraging hierarchical textual abstraction. While its performance is tied to the quality of initial MLLM descriptions, it effectively bridges the gap between raw visual data and high-level scene comprehension. Future work will explore adaptive thresholding for scene merging and more sophisticated "skimming" strategies for diverse video content.
Summary updated