Abstract
Video question answering benefits from the rich information in videos, enabling various applications. However, the large volume of tokens generated from long videos presents challenges to memory efficiency and model performance. To alleviate this, existing works propose to compress video inputs, but often overlook the varying importance of static and dynamic information across different queries, leading to inefficient token usage within limited budgets. We propose a novel token selection strategy, EXPLORE-THEN-SELECT (ETS), that adaptively adjusts static and dynamic information based on question requirements. Our framework first explores different token allocations between key frames, which preserve spatial details, and delta frames, which capture temporal changes. Then it employs a query-aware attention-based metric to select the optimal token combination without model updates. Our framework is plug-and-play and can be seamlessly integrated within diverse video language models. Extensive experiments show that our method achieves significant performance improvements (up to 5.8%) on multiple video question answering benchmarks.
1. Introduction
Video Large Language Models (VideoLMs) have demonstrated remarkable capabilities in understanding visual content, yet they face a significant "long-video problem." A single high-resolution frame can generate hundreds of tokens; processing a 5-minute video at standard sampling rates can produce tens of thousands of tokens, far exceeding the memory and compute limits of current hardware.
To address this, researchers typically compress videos via uniform sampling or pruning. However, these "one-size-fits-all" approaches ignore a crucial reality: different questions require different types of information. Identifying an object requires high-resolution spatial details (static information), while understanding an action sequence requires capturing temporal changes across consecutive frames (dynamic information). Existing methods often fail to balance these needs, leading to inefficient token usage.
2. Methodology: The EXPLORE-THEN-SELECT Framework
We propose the EXPLORE-THEN-SELECT (ETS) framework, a training-free, plug-and-play strategy that adaptively selects the best mix of static and dynamic video tokens based on the specific query.
2.1 Exploration: Search Space Design
Inspired by traditional video codecs like H.264, we partition the video into Key Frames (fully retained for spatial detail) and Delta Frames (sparse tokens capturing changes relative to key frames). The system generates multiple candidate token subsequences, each with a different ratio of key frames to delta frames, ranging from high-detail/low-motion to low-detail/high-motion, all while staying under a fixed token budget.
2.2 Selection: Query-Aware Attention
To select the optimal candidate without the cost of full model inference, we utilize the shallow layers of the VideoLM. We found that the second-layer attention provides a reliable "preview" of token importance. By passing the textual query and candidate visual tokens through just the first two layers, we calculate a Query-Aware Attention Score. The candidate that the query attends to most strongly is selected for final inference.
3. Experimental Results
We evaluated ETS across several rigorous benchmarks, including VideoMME (videos up to 1 hour), EgoSchema, and MLVU, using models such as Qwen2-VL and LLaVA-OneVision.
3.1 Performance Gains
ETS consistently outperformed standard uniform sampling and retrieval-based pruning methods. On the VideoMME benchmark, the framework improved Qwen2-VL-7B's performance by up to 5.8% on medium-length videos. The method is particularly effective at capturing subtle temporal changes that are critical for action-oriented questions.
3.2 Efficiency and Generalization
While the selection process adds a small amount of overhead (~0.5s first-token latency), it allows the model to process significantly more frames (e.g., sampling 256 frames but only processing 64) without exceeding GPU memory. Because it requires no training, it generalizes immediately across different model architectures.
4. Conclusion
The ETS framework demonstrates that query-adaptive token selection is essential for efficient long-video understanding. By leveraging shallow-layer attention as a selection metric, we achieve significant performance gains without the need for model retraining. Future work will focus on further reducing latency and improving compatibility with various positional encoding schemes.