Architectures for Streaming Video Understanding

620 concepts473 papers3 proposals

Most state-of-the-art video models operate under a comfortable assumption: they treat video as a static, pre-recorded file. In this offline processing paradigm, a model has access to the entire video sequence simultaneously. It can look ahead to future frames to resolve current ambiguities, perform global optimization passes over the entire timeline, and take its time generating a response after the video ends.

However, real-world applications like autonomous driving, robotics, and interactive AI assistants do not have access to the future. They operate in a streaming environment where data arrives sequentially, frame by frame, and decisions must be made immediately. Waiting for a video stream to conclude before processing it introduces unacceptable latency.

Shifting from offline batch processing to real-time streaming requires rethinking how models handle temporal context, manage memory, and execute inference under strict computational constraints.


The Core Challenge: The Memory and Latency Bottleneck

To understand why offline models fail in streaming scenarios, consider how a standard Vision-Large Language Model (VLLM) processes video. The model converts each video frame into a set of visual tokens and appends them to its context window.

As the video stream continues, the number of tokens grows linearly. Because the computational complexity of self-attention scales quadratically with context length, the model quickly encounters a severe memory bottleneck.

Offline Paradigm (Global Attention):
[ Frame 1 ] <---> [ Frame 2 ] <---> [ Frame 3 ] <---> [ Frame 4 ] ---> Full Context Available

Streaming Paradigm (Causal/Incremental):
[ Frame 1 ] ---> [ Frame 2 ] ---> [ Frame 3 ] ---> [ Frame 4 ] ---> Future Unknown
  (Past Memory Buffer) ----------> [ Current Frame ]

To maintain real-time throughput (e.g., processing at 10+ frames per second), streaming architectures must decouple their processing speed from the total duration of the video. They achieve this through two primary mechanisms: recurrent memory compression and dynamic token selection.

Recurrent and Persistent Memory

Instead of keeping every raw visual token in the active context window, streaming models compress historical frames into a fixed-size or bounded memory state.

For instance, Xiangyu Zeng et al. developed StreamForest: Efficient Online Video Understanding with Persistent Event Memory, which organizes visual data into a structured "Persistent Event Memory Forest." This architecture allows the model to maintain a long-term narrative thread while preserving high-resolution immediate perception for the current frame.

Similarly, in the context of highlight detection, Aiden Chang et al. introduced a framework called [Aha!

  • Predicting What Matters Next: Online Highlight Detection Without Looking Ahead](/work/aha-predicting-what-matters-next-online-highlight-detection-xnef), which utilizes a "Dynamic SinkCache" to maintain constant memory usage regardless of video length, enabling real-time autoregressive evaluation of streaming moments.

Attention-Driven Token Pruning

Not every pixel or frame contains novel information. In a streaming video, consecutive frames are highly redundant. Streaming architectures exploit this by using the model's internal state to dynamically prune uninformative tokens before they enter the heavy language-processing layers.

In Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs, Vaggelis Dorovatas et al. proposed a training-free method (rLiVS) that leverages the LLM's own attention scores from previous steps to filter out up to 95% of redundant visual tokens. By combining this selection mechanism with a recurrent memory of past clips, the system processes hour-long streams with bounded memory overhead.


Causal Temporal Modeling Without Future Context

In offline video tasks like segmentation or tracking, models often use bidirectional attention or global graph optimization to smooth out predictions. For example, if an object is occluded by a tree in frame 10 but reappears in frame 20, an offline model can look ahead to frame 20 to confirm the object's identity in frame 10.

In a streaming pipeline, the model must operate strictly causally. It must propagate its beliefs forward through time without retrospective smoothing.

Explicit Query Propagation

To track objects or segment regions in real time without multi-frame clip windows, models rely on sequential state propagation. In OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation, Dongming Wu demonstrated that a frame-by-frame approach can outperform complex clip-based methods by using an explicit query propagation mechanism. Instead of looking at a window of frames simultaneously, the model passes the object queries generated in the current frame directly to the next frame, treating the query itself as the temporal link.

Iterative Denoising and Prompts

This sequential dependency can also be framed probabilistically. For tasks like scene graph generation—where the model maps relationships between objects over time—the temporal context can act as a prior for current predictions.

Liulei Li et al. applied this concept to diffusion models in DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation. Their framework processes video frames sequentially by using the scene graph predictions from the previous frame as temporal "prompts" to guide the iterative denoising process of the current frame, bypassing the need for a global memory bottleneck.


Shifting from Static QA to Proactive Interaction

Traditional video question-answering (VideoQA) models follow a rigid "stop-and-process" routine: the user provides a video clip, asks a question, and the model generates a static response. A true streaming assistant, however, must interact with a continuous feed and decide when to act or speak without waiting for a user prompt or an end-of-video signal.

Traditional VideoQA:
[ Video Clip Ingestion ] ---> [ User Prompt ] ---> [ Model Response ]

Proactive Streaming:
[ Continuous Video Stream ] ---> [ Streaming EOS / Activation Model ] ---> [ Proactive Response ]
                                      (Model decides when to act)

Streaming Training Objectives

To train language models to handle continuous inputs, researchers have modified the standard autoregressive training objectives. In VideoLLM-online: Online Video Large Language Model for Streaming Video, Joya Chen et al. introduced the LIVE (Learning-In-Video-strEam) framework. Central to this approach is a "Streaming EOS" (End of Sequence) training objective, which teaches the model when to emit a response and when to remain silent and continue absorbing streaming visual inputs.

Decoupled Execution Pipelines

To prevent perception from blocking generation, streaming frameworks often decouple the visual processing pipeline from the cognitive decision-making pipeline.

Pan Zhang proposed this architectural split in Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. By disentangling perception, decision, and reaction into parallel threads, the model can continuously ingest streaming video frames while simultaneously executing language generation, eliminating the lag associated with traditional offline architectures.

Similarly, Haibo Wang et al. developed StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant, which uses a decoupled activation model alongside a smart memory compression strategy to bridge the gap between standard offline pre-trained models and the demands of live, proactive streaming environments.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_offline_processing_latency_30f916f7,
  title = {offline processing latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/offline-processing-latency/rdr_30f916f7b14440b5a6f79575cc14939f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_offline_processing_latency_30f916f7_v1,
  title = {offline processing latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/offline-processing-latency/rdr_30f916f7b14440b5a6f79575cc14939f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake