Introduction
Vision-Language Models (VLMs) have reached a milestone in video understanding, yet their practical application in real-time streaming remains limited by extreme computational demands. A standard video frame typically generates hundreds of visual tokens; for continuous streams, this leads to a "memory explosion" and prohibitive latency. VideoScan introduces an efficient inference framework that compresses each video frame into a single semantic carrier token. This method achieves a 5× speedup over state-of-the-art models like LLaVA-Video while maintaining stable GPU memory usage, regardless of video duration.
The VideoScan Framework
The core innovation of VideoScan is the decoupling of heavy vision processing from autoregressive text generation. It utilizes a two-phase approach to manage visual data:
- Prefilling Phase: Each frame is processed by a vision encoder. These features are average-pooled into a single "carrier" token. During this phase, the full visual tokens are processed once to populate the LLM's Key-Value (KV) cache and are then immediately discarded.
- Decoding Phase: The model generates responses using only the carrier tokens and their stored KV pairs, eliminating redundant visual computation.
Leveraging Semantic Flow
This extreme compression is made possible by Semantic Flow. Observations of Transformer decoders show that deeper layers naturally aggregate information from preceding tokens. By positioning the carrier token at the end of a frame's sequence, it "inherits" the summarized context of that frame, allowing the model to reason across time using highly compressed inputs.
Training and Memory Management
The model is trained in two stages: first, fine-tuning the vision encoder and LLM (via LoRA) to align with the pooled carrier representation; second, employing semantic-aware causal masking. This masking strategy forces the model to rely strictly on carrier tokens for temporal context, preventing information leakage. To ensure deployment on consumer-grade hardware, VideoScan implements a feature duplication-based eviction strategy that maintains a constant 18GB VRAM footprint by discarding redundant carrier tokens based on cosine similarity.
Experimental Results
VideoScan was evaluated on rigorous benchmarks including MLVU (multi-task) and VStream-QA (streaming). Key findings include:
- Efficiency: A reduction of vision tokens by over 99%.
- Speed: 6 FPS on an RTX 4090, representing a 5× improvement over LLaVA-Video.
- Accuracy: On the MLVU benchmark, VideoScan (61.3%) significantly outperformed other efficient models like LLaVA-Mini (42.8%) and LLaVA-VID (33.2%).
- Latency: On the VStream-QA benchmark, it reduced latency by 1.29× and VRAM usage by 50% compared to the ReKV baseline.
Conclusion and Future Work
VideoScan demonstrates that LLMs are remarkably resilient to visual data loss if the "semantic flow" is preserved. By reducing frames to single tokens, it enables "always-on" AI for robotics and AR assistants. While the current model may sacrifice some fine-grained spatial detail, future iterations could incorporate hybrid memory systems to selectively retain high-resolution tokens for tasks requiring extreme precision.
Summary updated