Vista: Scene-Aware Optimization for Streaming Video Question Answering Under Post-Hoc Queries

Haocheng Lu, Nan Zhang, Wei Tao, Xiaoyang Qu, Guokuan Li, Jiguang Wan, Jianzong Wang

TL;DR

Vista is a novel framework designed for "streaming" video question answering, where a model must process a continuous video feed and answer questions that arrive at unpredictable times. By dynamically grouping frames into semantically coherent "scenes," compressing them into compact tokens for GPU storage, and recalling high-resolution details only when relevant to a query, Vista achieves state-of-the-art performance while maintaining a tiny, stable memory footprint.

What Problem Does This Paper Address?

Most current Video Question Answering (Video QA) models operate in an "offline" mode: they are given a complete video and a question simultaneously. In the real world—think autonomous robots, security feeds, or live-stream assistants—video is a continuous, potentially infinite stream, and questions (post-hoc queries) can be asked at any moment.

This presents two massive hurdles:

  1. The Memory Wall: You cannot store every high-resolution frame of an infinite stream in expensive GPU memory without eventually crashing (OOM).
  2. The Latency Gap: Re-scanning hours of video every time a user asks a question is too slow for real-time interaction.

Existing solutions often use "sliding windows" (which forget the distant past) or "uniform sampling" (which might skip the exact moment needed to answer a specific question).

Methods

Vista (Scene-Aware Optimization) introduces a three-stage pipeline to handle streaming data efficiently:

  • Scene-aware Segmentation: Instead of treating video as a sequence of random frames, Vista uses an unsupervised online algorithm to detect "scene boundaries." It compares the current frame to both the previous frame and a fixed "anchor" frame. If the visual similarity drops below a threshold, it declares a new scene.
  • Scene-aware Compression: Once a scene is identified, it is compressed into a single compact scene token. This involves temporal averaging and a spatial weighting scheme (using L2 norms) to ensure salient visual features are preserved. These tiny tokens stay in the GPU for quick searching, while the original high-resolution frames are "offloaded" to the much larger (but slower) CPU RAM or disk.
  • Scene-aware Recall: When a user asks a question, the model doesn't look at everything. It compares the query to the stored scene tokens to find the top-$k$ most relevant historical scenes. It then "recalls" only those specific high-res frames from the CPU, combines them with the most recent "local window" of frames, and feeds them to a Multimodal Large Language Model (MLLM) to generate an answer.

Datasets

The authors evaluated Vista across several challenging benchmarks:

  • StreamingBench: A comprehensive suite specifically for streaming video understanding, covering real-time perception, emotion recognition, and long-term contextual reasoning.
  • MLVU & EgoSchema: Standard "offline" long-video benchmarks used to demonstrate that Vista’s streaming approach also excels at general long-form video understanding.

Key Insights and Ideas

  • Query-Agnostic Importance: Because the model doesn't know what the user will ask while the video is playing, it must organize memory based on visual coherence (scenes) rather than predicted relevance.
  • Hierarchical Memory Management: By treating the GPU as a "cache" for indices (tokens) and the CPU as a "warehouse" for data (frames), Vista breaks the linear relationship between video length and GPU memory usage.
  • Attention as a Retrieval Tool: The framework leverages the inherent attention mechanism of LLMs to perform "index-based retrieval," effectively turning the model's reasoning process into a search engine for its own memory.

Key Results

  • State-of-the-Art Performance: On StreamingBench, Vista (built on LLaVA-OneVision-7B) outperformed all other streaming models. Notably, in the "Multimodal Alignment" task, it achieved 74.00%, a staggering 29.6% absolute improvement over the baseline, even beating proprietary giants like GPT-4o (56.00%).
  • Efficiency & Scalability: While standard models saw their memory usage and latency (Time to First Token) skyrocket as video length increased, Vista’s resource consumption remained flat and stable even at 1,000+ frames.
  • Generalization: Vista improved the performance of different "backbone" models (like Video-LLaMA2), proving it is a model-agnostic framework that can upgrade existing AI architectures.

Limitations and Future Work

The authors identify a few areas for improvement:

  • Dynamic Scene Complexity: In videos with very rapid motion or "shaky cam," the unsupervised boundary detection can struggle, leading to fragmented scenes.
  • Static Scene Overload: Conversely, in very long, static shots (like a security camera), the model must enforce a maximum scene length to prevent memory overflow, which might artificially split a single semantic event.
  • Overlapping Events: If multiple important events happen simultaneously in different parts of the video, the current top-$k$ recall strategy might struggle to integrate all necessary context if $k$ is set too low.

Future work may involve more sophisticated, learnable scene segmentation or adaptive $k$-selection based on query complexity.

Summary updated

Abstract

Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.
BibTeX
@inproceedings{lu2026vista,
  author = {Haocheng Lu and Nan Zhang and Wei Tao and Xiaoyang Qu and Guokuan Li and Jiguang Wan and Jianzong Wang},
  title = {{Vista: Scene-Aware Optimization for Streaming Video Question Answering Under Post-Hoc Queries}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i9.37694},
}

Loading related content...

Figure