Abstract
Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.
Introduction
Most current Video Question Answering (Video QA) models operate in an "offline" mode, where a complete video and a question are provided simultaneously. However, real-world applications—such as autonomous robotics, security monitoring, or live-stream assistants—require processing continuous, potentially infinite video streams where questions (post-hoc queries) can arrive at any moment.
This streaming paradigm introduces two primary challenges:
- The Memory Wall: Storing every high-resolution frame of an infinite stream in GPU memory is impossible, eventually leading to Out-of-Memory (OOM) errors.
- The Latency Gap: Re-processing hours of video for every new query is computationally prohibitive for real-time interaction.
Existing approaches like sliding windows or uniform sampling often fail by either losing distant context or missing critical, fine-grained details necessary for specific queries.
The Vista Framework
Vista (Scene-Aware Optimization) addresses these challenges through a three-stage pipeline designed to handle streaming data efficiently without prior knowledge of the user's queries.
1. Scene-aware Segmentation
Instead of arbitrary frame grouping, Vista employs an unsupervised online algorithm to detect "scene boundaries." By comparing the current frame to both its immediate predecessor and a fixed "anchor" frame within the scene, Vista identifies visual shifts. When similarity drops below a threshold, a new scene is initialized, ensuring that the video is organized into semantically coherent units.
2. Scene-aware Compression
Once a scene is segmented, it is compressed into a single compact scene token. This process utilizes temporal averaging combined with a spatial weighting scheme (based on L2 norms) to preserve salient visual features. These lightweight tokens are stored in GPU memory to facilitate rapid searching. Meanwhile, the original high-resolution frames are offloaded to larger, more cost-effective storage like CPU RAM or disk.
3. Scene-aware Recall
Upon receiving a query, Vista performs "index-based retrieval." It compares the query embedding against the stored scene tokens to identify the top-$k$ most relevant historical scenes. The system then "recalls" only the high-resolution frames associated with these specific scenes from the CPU. These recalled frames are combined with a "local window" of the most recent frames and fed into a Multimodal Large Language Model (MLLM) for final answer generation.
Experimental Evaluation
The authors evaluated Vista using StreamingBench, a comprehensive benchmark for real-time perception and long-term reasoning, as well as standard long-video benchmarks like MLVU and EgoSchema.
State-of-the-Art Performance
On StreamingBench, Vista (integrated with LLaVA-OneVision-7B) achieved significant performance gains. In the "Multimodal Alignment" task, it reached 74.00%, representing a 29.6% absolute improvement over the baseline and outperforming proprietary models like GPT-4o.
Efficiency and Scalability
A key advantage of Vista is its resource stability. As shown in the performance analysis, while standard "All Frames" models quickly hit memory limits or suffer from exponential latency increases, Vista's resource footprint remains flat and predictable even as video length scales.
Vista breaks the linear relationship between video length and GPU memory usage, allowing it to process over 1,000 frames with minimal impact on Time to First Token (TTFT), whereas traditional methods encounter OOM errors.
Discussion and Limitations
Vista demonstrates that query-agnostic memory organization based on visual coherence is a powerful strategy for streaming AI. By treating the GPU as a high-speed cache for indices and the CPU as a warehouse for raw data, Vista enables long-context reasoning with high efficiency.
However, some limitations remain. Unsupervised boundary detection can be sensitive to rapid camera motion or "shaky cam," potentially leading to fragmented scenes. Additionally, in extremely long static shots, the model must enforce maximum scene lengths to prevent local memory issues, which may split single semantic events. Future work will explore adaptive recall strategies and more robust, learnable segmentation methods to further enhance real-world applicability.
Summary updated