Processing video sequences requires capturing how objects, motion, and context evolve over time. Traditional video models typically rely on two paradigms: sliding windows (such as 3D CNNs) that look at a fixed number of neighboring frames, or global self-attention in Transformers that computes relationships across all frames simultaneously.
However, both approaches hit fundamental scaling walls. Sliding windows are inherently short-sighted, resetting at segment boundaries and discarding long-range dependencies. Conversely, global self-attention suffers from a "token explosion"—as video length grows, the computational and memory costs scale quadratically, quickly exceeding hardware limits.
To overcome these limitations, a powerful paradigm has emerged across video understanding, generation, and compression: recurrent hidden state propagation. Instead of keeping all historical frames or tokens explicitly in the active context window, these models compress past information into a persistent, evolving latent representation—a hidden state, a state-space vector, or a set of learnable memory tokens. This state is passed sequentially from one frame or video segment to the next, decoupling the model's temporal receptive field from its immediate computational footprint.
The Core Mechanism: Latent State Continuity
The foundational intuition behind recurrent propagation is that a model does not need to re-examine raw past frames to understand the present; it only needs a well-distilled summary of the past. By passing a persistent hidden state between time steps, a network can accumulate information over an indefinite sequence length.
[Frame t-1] ----> [ Encoder ]
│
▼
[Hidden State t-1] ──►(Update)──► [Hidden State t] ──► [ Decoder ] ──► [Output t]
▲
│
[Frame t] ----> [ Encoder ]
In low-level video processing, this mechanism provides a highly efficient alternative to bulky spatio-temporal convolutions. For video super-resolution, Takashi Isobe et al. introduced the Recurrent Residual Network in Revisiting Temporal Modeling for Video Super-resolution. They demonstrated that passing a persistent hidden state stabilized by residual connections allows the network to aggregate fluid temporal context over long sequences, achieving state-of-the-art results while being significantly faster and more parameter-efficient than 3D CNN counterparts.
Similarly, maintaining temporal continuity is vital for tracking high-frequency motion without jitter. In Capturing Spatial and Temporal Patterns for Facial Landmark Tracking through Adversarial Learning, Shangfei Wang et al. showed that propagating hidden states across consecutive temporal windows ensures that the model retains structural and rhythmic context throughout the entire duration of a video, rather than resetting or glitching at segment boundaries.
Adapting Recurrence to Transformers and World Models
While classical recurrence handles frame-by-frame processing well, modern architectures like Transformers and Diffusion Models require novel ways to integrate recurrent states without sacrificing their representation power.
Recurrent Attention Gates and Windows
Rather than computing global attention maps across an entire video, models can process data in localized windows or frames while using recurrent mechanisms to bridge the temporal gaps. In Flow-Guided Sparse Transformer for Video Deblurring, Jing Lin et al. integrated a hidden state mechanism directly into a transformer architecture. This allows the model to maintain a persistent representation of the scene across long sequences, using optical flow to guide sparse self-attention patches without the memory overhead of global frame attention.
For high-level action recognition, Jiewen Yang et al. proposed the Recurrent Vision Transformer (RViT) in Recurring the Transformer for Video Action Recognition. RViT processes videos frame-by-frame, employing a novel attention gate to pass temporal information through a hidden state. This architecture captures extended actions spanning long durations while using up to 10× less GPU memory than traditional clip-based transformers.
To maintain structural continuity in tracking, Shi Yin et al. developed 1DFormer: A Transformer Architecture Learning 1D Landmark Representations for Facial Landmark Tracking. Their framework utilizes a recurrent window-based processing strategy that passes latent state information between successive sliding windows, integrating long-range historical context without requiring the entire video sequence to reside in memory simultaneously.
Learnable Memory Tokens
To avoid the gradient vanishing or exploding issues historically associated with traditional RNNs, several architectures leverage learnable memory tokens that act as a dynamic scratchpad passed between sequential video clips.
[Clip 1] ──► [Transformer Block] ──► [Updated Memory Token]
│
▼
[Clip 2] ──► [Transformer Block] ──► [Updated Memory Token]
▲
│
(Attends to Memory Token)
In Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs, Jingfei Xia et al. introduced a Memory Recurrent Unit (MRU) that passes a learnable memory token between segmented video clips. This provides a stable way to aggregate global performance information over extended sports routines with linear computational complexity.
This token-based propagation is equally powerful for predictive tasks. In Early Anticipation of Driving Maneuvers, Abdul Wasi et al. utilized learnable memory tokens to retain context across long multi-view video streams. This mechanism allows the model to relate historical driver behavior to current observations and anticipate maneuvers seconds before they occur, without needing to process the full history simultaneously.
Streaming Long Video Understanding
When dealing with hour-long videos or continuous streams, loading all frames into an LLM's context window is impossible. To solve this, Rui Qian et al. developed a streaming framework detailed in Streaming Long Video Understanding with Large Language Models. By processing video in sequential segments where the encoding of the current clip incorporates a compressed representation of all previous segments, the authors achieved a constant memory footprint capable of handling arbitrary video lengths.
Taking a question-aware approach, Di Hu et al. introduced Video Detective: Seek Critical Clues Recurrently to Answer Question from Long Videos. This architecture recurrently aggregates and updates a small set of memory tokens as the video plays, allowing Multi-modal LLMs to "think while watching" and process massive video inputs in minutes on standard hardware.
Linear-Time State Space Models
The paradigm also extends to generative world models used in reinforcement learning. In EDELINE: Enhancing Memory in Diffusion-based World Models via Linear-Time Sequence Modeling, Jia-Hua Lee et al. integrated Mamba State Space Models (SSMs) into diffusion-based world models. By replacing fixed-frame context windows with a linear-time recurrent state space, the model allows agents to maintain temporal consistency and retain critical environmental information—such as map layouts or object locations—over long sequences.
Implicit Reference Buffers in Video Compression
Neural video compression is another domain where recurrent hidden state propagation fundamentally changes how models operate. Traditional video codecs rely on explicit, short-term reference frames to compute motion compensation and residuals. If an object is occluded and reappears ten frames later, the codec must transmit new data because the immediate reference frame no longer contains it.
By integrating recurrent layers into the compression pipeline, the network effectively creates an implicit, long-term reference buffer that accumulates historical context over an extended duration.
Traditional Codec: [Frame t-2] ──► [Frame t-1] ──► [Frame t] (Short-term reference only)
Recurrent Codec: [Frame t-2] ──► [Frame t-1] ──► [Frame t]
│ │ │
▼ ▼ ▼
[ Latent ───►───► Latent ───►───► Latent ] (Accumulated History Prior)
In Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model, Fabian Mentzer et al. integrated recurrent cells into both the auto-encoder and the probability model of a video codec. This architecture exploits long-range temporal correlations across many frames, creating a rich temporal prior that significantly improves compression efficiency.
Building on this concept, Ren Yang et al. developed Advanced Learned Video Compression (ALVC) in Advancing Learned Video Compression With In-Loop Frame Prediction. ALVC uses recurrent layers to maintain a long-term memory of motion context, enabling the model to predict subsequent frames from historical data at zero bit-rate cost, drastically reducing the data needed to correct motion errors.
This latent propagation addresses a critical limitation noted by Abdelaziz Djelouah et al. in Neural Inter-Frame Compression for Video Coding, where standard interpolation-based frameworks limit their scope to immediate frame pairs and ignore broader temporal context.
Furthermore, Jinxi Xiang et al. demonstrated in MIMT: Masked Image Modeling Transformer for Video Compression that integrating recurrent hidden states into a transformer-based entropy model provides a superior temporal prior compared to rigid raster-scan models, leveraging long-range dependencies to achieve substantial bitrate savings.
Why It Works: The Intuition of Constant-Space Context
The success of recurrent hidden state propagation across these diverse tasks rests on three core architectural principles:
- Information Compression as a Bottleneck: Instead of forcing a model to attend to every raw pixel or token from the past (which causes quadratic complexity), the recurrent update step acts as an information bottleneck. It forces the network to discard redundant spatial details (like compression artifacts or background noise) while distilling and updating a compact latent summary. For example, in Learning Spatiotemporal Frequency-Transformer for Compressed Video Super-Resolution, Zhongwei Qiu et al. showed that propagating a hidden state in the frequency domain allows the model to separate true visual textures from compression artifacts over long sequences.
- $O(1)$ Memory Scaling: By passing a fixed-size hidden state or a bounded set of memory tokens across sliding windows or sequential frames, the model achieves an effectively infinite temporal receptive field. The memory footprint remains constant ($O(1)$) relative to video length, preventing the token explosion that cripples standard transformers on long-form content.
- Elimination of Boundary Artifacts: Traditional chunk-based or sliding-window processing suffers from temporal boundary discontinuities because the model's memory is wiped clean at the start of each new segment. Continuous state propagation ensures that the features of the current frame are always conditioned on the entire history of the sequence, preserving global context and long-term consistency.
Key Foundational Work
Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model (2020) — contributed 1 concept elements to this direction.
Revisiting Temporal Modeling for Video Super-resolution (2020) — contributed 1 concept elements to this direction.
Capturing Spatial and Temporal Patterns for Facial Landmark Tracking through Adversarial Learning (2019) — contributed 1 concept elements to this direction.