Traditional video models treat video sequences as static 3D volumes or process them in large, multi-frame clips. While effective for offline analysis, this approach introduces a fundamental computational bottleneck: memory and latency scale linearly ($O(T)$) with the length of the video. For real-time applications like autonomous flight, live clinical monitoring, or infinite-length video generation, an $O(T)$ dependency inevitably leads to hardware failure or unacceptable latency.
To solve this, a prominent research direction focuses on achieving online-offline performance parity. The core insight is that we do not need to look into the future, nor do we need to retain raw historical frames. Instead, by processing videos sequentially or in small, fixed-size temporal windows ($O(1)$ complexity) and propagating compressed historical context forward through specialized memory structures, online models can match or even exceed the accuracy of global offline models.
Bounding the Temporal Window
To achieve constant-time latency and memory consumption, a model must decouple its processing footprint from the total duration of the video. This is accomplished by shifting from global sequence attention to streaming windows.
In Streaming Video Temporal Action Segmentation In Real Time, Wen et al. demonstrate that processing video in fixed-size chunks shifts the computational complexity of temporal action segmentation to a constant-time latency baseline. This allows models to run indefinitely on live streams without accumulating lag. This principle applies equally to early video representation; Xu et al. show in LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing that streaming spatiotemporal processing over small, overlapping windows yields segmentation boundaries and temporal consistency nearly identical to processing the entire video globally.
This windowing logic extends beyond video analysis into generative modeling. In DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes, Liu et al. utilize discrete spatial and temporal windows within a video diffusion framework. By bounding the denoising process to local windows rather than a global volume, they generate infinite-length looping panoramic videos while maintaining a strictly constant VRAM footprint.
Compressing the Past via Latent Query Propagation
If a model only observes a narrow temporal window, it risks losing track of objects during occlusions or suffering from feature drift. Online architectures overcome this by compressing historical context into compact, dynamically updated latent representations rather than storing raw frames.
[Frame t-1] ───► [Memory Update] ───► [Updated Memory Bank]
▲ │
│ ▼
[Frame t ] ───► [Feature Extraction] ──► [Online Prediction]
This mechanism is highly effective for tracking and instance segmentation:
- Explicit Query Propagation: In OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation, Wu et al. challenge the necessity of clip-based offline modeling by showing that explicitly propagating object queries frame-by-frame acts as a highly efficient temporal anchor, outperforming complex offline baselines.
- Working Memory Decoders: In InstanceFormer: An Online Video Instance Segmentation Framework, Koner et al. employ a specialized working memory decoder to track instances across challenging sequences, proving that global visibility is unnecessary when instance identities are maintained recurrently.
- Memory Tokens and Codes: Hwang et al. introduce Video Instance Segmentation using Inter-Frame Communication Transformers, which uses isolated "memory tokens" to summarize and communicate information across frames at near-real-time speeds (89.4 FPS). Similarly, Li et al. show in Hybrid Instance-Aware Temporal Fusion for Online Video Instance Segmentation that condensing objects into global "instance codes" allows online transformers to match the precision of full-sequence offline models.
- Targeted Point Tracking: For long-term point tracking, Aydemir et al. develop Track-On: Transformer-based Online Point Tracking with Memory. By utilizing dual memory modules to manage historical point appearances, the model tracks points through long occlusions without ever looking at future frames.
Short-Term Alignment and Temporal Consistency
A common failure mode of frame-by-frame processing is temporal flickering or jitter, caused by slight variations in per-frame feature extractions. To enforce smoothness, online models pair long-term memory banks with short-term spatial alignment mechanisms.
In autonomous flight settings where low latency is critical, Vincent et al. introduce High Temporal Consistency through Semantic Similarity Propagation in Semi-Supervised Video Semantic Segmentation for Autonomous Flight. They combine feature similarity propagation with global camera alignment, yielding a 12.5% increase in prediction stability while maintaining the high inference speeds required for edge deployment.
In the medical domain, where rapid camera movement can obscure targets, Hu et al. design SALI: Short-term Alignment and Long-term Interaction Network for Colonoscopy Video Polyp Segmentation. SALI uses a localized spatial alignment module to bridge adjacent frames while interacting with a long-term memory bank of past polyp appearances, ensuring stable tracking through rapid camera shifts.
This integration of local alignment and memory also scales to low-level motion estimation. Liu et al. introduce ARFlow: Auto-regressive Optical Flow Estimation for Arbitrary-Length Videos via Progressive Next-Frame Forecasting, an auto-regressive framework that forecasts the next frame's motion based on a sliding-window memory of past estimates. This allows the model to process arbitrarily long videos with a fixed memory footprint of just 2.1GB. The foundational math of this paradigm traces back to structural modeling; Pirsiavash et al. demonstrated in Parsing Videos of Actions with Segmental Grammars that combining regular grammars with finite-state machines enables linear-time, constant-memory parsing of continuous action streams.
Why Incremental Decisions Can Outperform Global Context
It is highly intuitive to assume that a model with access to the entire video sequence (both past and future) will inherently outperform a sequential model. However, empirical evidence across this research direction challenges this assumption.
Global offline models often suffer from optimization difficulties over long temporal horizons, feature dilution, and high computational overhead that forces researchers to downsample spatial resolution or clip lengths. Online models avoid these pitfalls by focusing computational capacity entirely on local, incremental decision-making.
For example, in Online Multi-modal Person Search in Videos, Rao et al. combine multi-modal features with a reinforcement learning-driven memory bank and an uncertainty cache. This incremental search engine actually outperforms traditional offline methods that process the entire dataset globally, proving that local decision-making with uncertainty management can be more robust than global matching. Similarly, Elqursh et al. demonstrate in Online Motion Segmentation Using Dynamic Label Propagation that treating motion segmentation as an incremental manifold separation problem yields real-time results within a tight margin of offline state-of-the-art methods, while providing massive computational speedups.
Crucially, this online efficiency can be engineered so that multi-frame overhead is completely eliminated during deployment. In Is It Necessary to Transfer Temporal Knowledge for Domain Adaptive Video Semantic Segmentation?, Wu et al. show that enforcing temporal consistency constraints during training allows a model to learn highly stable representations. At test time, the model performs pure per-frame inference without requiring optical flow or multi-frame lookups, achieving superior domain adaptation accuracy while running at maximum hardware efficiency.
Key Foundational Work
Parsing Videos of Actions with Segmental Grammars (2014) — contributed 1 concept elements to this direction.
Online Motion Segmentation Using Dynamic Label Propagation (2013) — contributed 1 concept elements to this direction.