Enhancing Long Video Question Answering with Scene-Localized Frame Grouping

Xuyi Yang, Wenhao Zhang, Hongbo Jin, Lin Liu, Hongbo Xu, Yongwei Nie, Fei Yu, Fei Ma

Abstract

Multimodal Large Language Models (MLLMs) face significant challenges when processing long-form video content due to the "resource wall" that prevents simultaneous processing of all frames. This paper introduces Scene-Localized Frame Grouping (SLFG), a plug-and-play framework that organizes video frames into semantically coherent "scenes" to facilitate better reasoning. Alongside SLFG, the authors present SceneQA, a new task for scene-level comprehension, and LVSQA, a benchmark dataset of videos exceeding 30 minutes. SLFG demonstrates substantial performance gains across multiple long-video benchmarks without requiring model retraining.

Introduction

Most existing Video Question Answering (VideoQA) benchmarks focus on "needle-in-a-haystack" retrieval, asking models to locate specific objects in single frames. However, true video understanding requires a "scene-level" grasp of actions and relationships over time. Current MLLMs often downsample long videos so aggressively that they lose the fine-grained details necessary for complex narrative reasoning.

Comparison between traditional frame-level VideoQA and the proposed scene-level SceneQA task.

The SLFG Framework

The proposed SLFG method mimics human cognition by treating the "scene" as the primary unit of analysis. It operates through a four-stage pipeline designed to bridge the gap between raw visual data and high-level reasoning:

  1. Group Frames Description: The video is densely sampled and grouped, with a base MLLM generating textual descriptions for each group.
  2. Scene Generation: An LLM (e.g., Qwen2.5-7B) abstracts these descriptions into concise scene summaries, reducing noise and redundancy.
  3. Scene Localization: A text embedding model (BGE-M3) identifies the most relevant scenes by comparing summaries to the user's question.
  4. Dynamic Frame Grouping: The system merges consecutive high-scoring segments and expands the temporal window to ensure the model captures the full context of an event rather than just a snapshot.
The four-stage SLFG pipeline: from frame grouping and description to scene localization and dynamic reorganization.

Experimental Results and Benchmarks

The authors introduced the LVSQA (Long Video Scene-level Question Answering) dataset, featuring 100 videos (30+ minutes each) and 500 human-refined QA pairs focused on detail recognition and causal reasoning. In evaluations, SLFG consistently enhanced state-of-the-art models:

  • LVSQA: LLaVA-Video + SLFG reached 63.4% accuracy, a significant improvement over the baseline.
  • LVBench: LLaVA-OneVision achieved a 9.8% relative increase in accuracy when using the SLFG framework.
  • Efficiency: The marginal cost per question decreases significantly in multi-query scenarios, as the visual preprocessing and scene generation stages are performed only once per video.

Conclusion and Future Work

SLFG offers a scalable, training-free solution for long-video understanding by leveraging hierarchical textual abstraction. While its performance is tied to the quality of initial MLLM descriptions, it effectively bridges the gap between raw visual data and high-level scene comprehension. Future work will explore adaptive thresholding for scene merging and more sophisticated "skimming" strategies for diverse video content.

Summary updated

Abstract

Current Multimodal Large Language Models (MLLMs) often perform poorly in long video understanding, primarily due to resource limitations that prevent them from processing all video frames and their associated information. Efficiently extracting relevant information becomes a challenging task. Existing frameworks and evaluation tasks focus on identifying specific frames containing core objects from a large number of irrelevant frames, which does not align with the practical needs of real-world applications. To address this issue, we propose a new scenario under the video question-answering task, SceneQA, which emphasizes scene-based detail perception and reasoning abilities. And we develop the LVSQA dataset to support the SceneQA task, which is built upon carefully selected videos from LVBench and contains a new collection of question-answer pairs to promote a more fair evaluation of MLLMs' scene perception abilities in long videos. Inspired by human cognition, we introduce a novel method called SLFG. The core idea of SLFG is to combine individual frames into semantically coherent scene frames. By leveraging scene localization methods and dynamic frame reassembly mechanisms, SLFG significantly enhances the understanding capabilities of existing MLLMs in long videos. SLFG requires no modification to the original model architecture and boasts excellent plug-and-play usability. Experimental results show that this method performs exceptionally well in several long video benchmark tests. Code and dataset will be released at http://www.slfg.pkuzwh.cn.
BibTeX
@misc{yang2025enhancing,
  author = {Xuyi Yang and Wenhao Zhang and Hongbo Jin and Lin Liu and Hongbo Xu and Yongwei Nie and Fei Yu and Fei Ma},
  title = {{Enhancing Long Video Question Answering with Scene-Localized Frame Grouping}},
  year = {2025},
  eprint = {2508.03009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.03009},
}
Figure