VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers

Ruanjun Li, Yuedong Tan, Yuanming Shi, Jiawei ShaoarXiv 2025 · DOI · arXiv

Introduction

Vision-Language Models (VLMs) have reached a milestone in video understanding, yet their practical application in real-time streaming remains limited by extreme computational demands. A standard video frame typically generates hundreds of visual tokens; for continuous streams, this leads to a "memory explosion" and prohibitive latency. VideoScan introduces an efficient inference framework that compresses each video frame into a single semantic carrier token. This method achieves a 5× speedup over state-of-the-art models like LLaVA-Video while maintaining stable GPU memory usage, regardless of video duration.

Figure 1: VideoScan achieves a >99% reduction in vision tokens while maintaining high accuracy compared to state-of-the-art video VLMs.

The VideoScan Framework

The core innovation of VideoScan is the decoupling of heavy vision processing from autoregressive text generation. It utilizes a two-phase approach to manage visual data:

  1. Prefilling Phase: Each frame is processed by a vision encoder. These features are average-pooled into a single "carrier" token. During this phase, the full visual tokens are processed once to populate the LLM's Key-Value (KV) cache and are then immediately discarded.
  2. Decoding Phase: The model generates responses using only the carrier tokens and their stored KV pairs, eliminating redundant visual computation.

Leveraging Semantic Flow

This extreme compression is made possible by Semantic Flow. Observations of Transformer decoders show that deeper layers naturally aggregate information from preceding tokens. By positioning the carrier token at the end of a frame's sequence, it "inherits" the summarized context of that frame, allowing the model to reason across time using highly compressed inputs.

Figure 4: Attention visualization at Layer 28 shows how the model aggregates visual information, justifying the use of a compressed semantic carrier.

Training and Memory Management

The model is trained in two stages: first, fine-tuning the vision encoder and LLM (via LoRA) to align with the pooled carrier representation; second, employing semantic-aware causal masking. This masking strategy forces the model to rely strictly on carrier tokens for temporal context, preventing information leakage. To ensure deployment on consumer-grade hardware, VideoScan implements a feature duplication-based eviction strategy that maintains a constant 18GB VRAM footprint by discarding redundant carrier tokens based on cosine similarity.

Experimental Results

VideoScan was evaluated on rigorous benchmarks including MLVU (multi-task) and VStream-QA (streaming). Key findings include:

  • Efficiency: A reduction of vision tokens by over 99%.
  • Speed: 6 FPS on an RTX 4090, representing a 5× improvement over LLaVA-Video.
  • Accuracy: On the MLVU benchmark, VideoScan (61.3%) significantly outperformed other efficient models like LLaVA-Mini (42.8%) and LLaVA-VID (33.2%).
  • Latency: On the VStream-QA benchmark, it reduced latency by 1.29× and VRAM usage by 50% compared to the ReKV baseline.

Conclusion and Future Work

VideoScan demonstrates that LLMs are remarkably resilient to visual data loss if the "semantic flow" is preserved. By reducing frames to single tokens, it enables "always-on" AI for robotics and AR assistants. While the current model may sacrifice some fine-grained spatial detail, future iterations could incorporate hybrid memory systems to selectively retain high-resolution tokens for tasks requiring extreme precision.

Summary updated

Abstract

This paper introduces VideoScan, an efficient vision-language model (VLM) inference framework designed for real-time video interaction that effectively comprehends and retains streamed video inputs while delivering rapid and accurate responses. A longstanding challenge in video understanding--particularly for long-term or real-time applications--stems from the substantial computational overhead caused by the extensive length of visual tokens. To address this, VideoScan employs a single semantic carrier token to represent each frame, progressively reducing computational and memory overhead during its two-phase inference process: prefilling and decoding. The embedding of the semantic carrier token is derived from an optimized aggregation of frame-level visual features, ensuring compact yet semantically rich representations. Critically, the corresponding key-value pairs are trained to retain contextual semantics from prior frames, enabling efficient memory management without sacrificing temporal coherence. During inference, the visual tokens of each frame are processed only once during the prefilling phase and subsequently discarded in the decoding stage, eliminating redundant computations. This design ensures efficient VLM inference even under stringent real-time constraints. Comprehensive experiments on diverse offline and online benchmarks demonstrate that LLaVA-Video, supported by our method, achieves up to $\sim 5\times$ and $1.29\times$ speedups compared to its original version and previous efficient streaming video understanding approaches, respectively. Crucially, these improvements are attained while maintaining competitive performance and ensuring stable GPU memory consumption (consistently $\sim 18$GB, independent of video duration).
BibTeX
@misc{li2025videoscan,
  author = {Ruanjun Li and Yuedong Tan and Yuanming Shi and Jiawei Shao},
  title = {{VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers}},
  year = {2025},
  eprint = {2503.09387},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.09387},
}
Figure