Large Multimodal Models (LMMs) excel at describing a single image, but they struggle when faced with a one-hour video. The reason is a "token explosion": a standard vision transformer might represent a single frame with 576 tokens. At 30 frames per second, a one-hour video generates over 60 million tokens—far exceeding the context window and VRAM capacity of even the most powerful GPUs.
To bridge this gap, researchers are developing strategies to bypass this "memory wall" by treating video not as a dense sequence of images, but as a sparse, compressible, and retrievable stream of information.
The Efficiency of Extreme Compression
The most direct way to handle more video is to make each frame "smaller" in the eyes of the language model. Standard architectures often treat every patch of an image as equally important, but for video, much of this data is redundant.
Chengyao Wang and colleagues introduced LLaMA-VID, which reduces a frame to just two tokens: a context token for the overall scene and a content token for visual details. This massive reduction allows models to ingest hour-long content without saturating the context window. Similarly, Matryoshka Multimodal Models by Mu Cai and Jianwei Yang use a "nested" approach, allowing the model to dynamically choose between high-detail and low-detail representations depending on the task's complexity.
Query-Aware Sparsity
Not every second of a video is relevant to a user's question. If you ask, "When did the red car crash?", the model doesn't need high-resolution tokens for the ten minutes of highway driving that preceded it.
Recent work focuses on Query-Aware Selection, where the model uses the text prompt to filter visual data before it ever reaches the heavy reasoning layers:
- Frame Selection: Q-Frame and APVR identify "pivot" frames that are semantically related to the query, discarding the rest.
- Token Pruning: Siyou Li proposed QTSplus, which compresses the vision stream by up to 89% by selecting only the specific visual tokens within a frame that answer the query.
- Dynamic Depth: Shiwei Wu and Joya Chen developed VideoLLM-MoD, which uses a "Mixture-of-Depths" approach to skip heavy computations for redundant tokens, focusing resources only on the most salient visual cues.
Retrieval-Augmented Video Understanding
When a video is too long to fit in memory even with compression, researchers turn to Retrieval-Augmented Generation (RAG). Instead of feeding the entire video into the model, the system treats the video as a database.
Shukang Yin and the team behind Video-RAG demonstrate that by retrieving only the most visually aligned segments for a given query, even modest open-source models can understand hour-long documentaries. OneClip-RAG further refines this by retrieving semantically coherent "clips" rather than isolated frames, preserving the temporal flow necessary for complex reasoning.
Streaming and Recurrent Memory
For "always-on" or streaming applications, the model cannot wait for the video to end to begin processing. VideoLLM-online introduces a streaming framework that maintains a processing speed of over 10 FPS by treating video as a continuous flow.
To prevent memory overflow in these scenarios, Di Hu and others developed Video Detective, which uses a recurrent memory mechanism. It "thinks while watching," compressing thousands of frames into a small set of evolving "memory tokens." This allows the model to maintain a long-term narrative summary without storing every raw visual token in the Key-Value (KV) cache.
The "Free Lunch" of Sparse Decoding
Perhaps the most promising direction is the discovery of "plug-and-play" acceleration. Qian Liu and Xuan Zhang identified a "free lunch" in Sparse-to-Dense (STD). By exploiting the fact that models naturally pay attention to only a few frames during decoding, they achieve nearly 2x speedups without any additional training or loss in accuracy.
By combining these techniques—compression, query-aware filtering, and RAG—the research community is moving toward a future where "hour-scale" video understanding is no longer a hardware impossibility, but a standard capability of multimodal AI.