Scaling Video Transformers via Spatiotemporal Sparsity

This direction is split into other directions: quadratic attention complexity (quadratic attention scaling). Its last version stays readable and citable here.

Version 1 of 1522 papers553 conceptsfrozen 2026-10-04

The central challenge in modern video AI is the "token explosion." While a single image might be decomposed into a few hundred patches, a one-minute high-definition video generates hundreds of thousands of tokens. Because standard self-attention complexity scales quadratically ($O(N^2)$), doubling the video length quadruples the computational cost. This bottleneck makes high-resolution, long-duration video generation and understanding prohibitively expensive on current hardware.

To bypass this "quadratic wall," researchers are moving away from dense attention toward mechanisms that exploit the inherent redundancy of video data.

The Redundancy of the Video Cube

In a video sequence, most tokens are redundant. A pixel in frame 10 is likely very similar to the same pixel in frame 11, and large regions of the background may remain static for seconds. Dense attention ignores this, treating every token-to-token relationship as equally important.

Recent work focuses on Structured Sparsity, which restricts attention to the most informative connections. Qirui Li and colleagues in Compact Attention exploit predictable patterns like local clusters and "cross-shaped" interactions, achieving significant speedups by ignoring distant, irrelevant tokens. Similarly, RainFusion2.0, developed by Aiyue Chen and others, uses block-wise representations and spatiotemporal windowing to achieve 80% sparsity with almost no loss in visual quality.

Dynamic and Learnable Routing

Static sparsity (like fixed windows) can miss fast-moving objects or long-range dependencies. To solve this, researchers are developing Dynamic Sparsity—mechanisms that decide which tokens to attend to on the fly.

Xin Tan and the DSV team identify "critical" attention connections dynamically during training, allowing the model to focus its budget on high-entropy regions. Taking this further, Yuwei Guo introduced Mixture of Contexts (MoC), which treats long-video generation as an information retrieval problem. MoC uses a learnable router to send queries only to the most relevant "chunks" of past video data, resulting in near-linear scaling.

Linearization and Hybrid Architectures

Another strategy is to replace the attention mechanism entirely with operations that scale linearly ($O(N)$). This is often achieved by converting the "quadratic" self-attention into a recurrent or state-space representation.

In Attention Surgery, Denis Korzhenkov and Amir Habibian demonstrate a recipe for "linearizing" Diffusion Transformers (DiTs). By distilling a pre-trained quadratic model into a hybrid version, they achieve a 6x speedup for long videos. Junsong Chen and Song Han take this further with SANA-Video, which uses a Linear DiT and a constant-memory KV cache to generate minute-long HD video on consumer-grade hardware.

Token Compression and Informational Uniqueness

If the number of tokens ($N$) is the problem, one solution is to simply have fewer tokens. Token Compression identifies and merges or prunes redundant visual information before it reaches the expensive attention layers.

Can Qin and the DyCoke team argue that static pruning is insufficient; they propose dynamic compression that adapts to the video content. Similarly, UniComp by Limeng Qiao prioritizes "informational uniqueness," mathematically linking a token's novelty to its importance. This allows models to retain as little as 10% of visual tokens while maintaining high accuracy in video understanding tasks.

Hardware-Aware Implementation

A major hurdle for sparse attention is that "theoretical" FLOP reductions often don't translate to real-world speedups because GPUs are optimized for dense matrix multiplication.

To bridge this gap, researchers are designing Hardware-Aware Sparse Attention. Sankeerth Durvasula and the FG-Attn team introduced a custom GPU primitive called "asynchronous-gather load." This allows the hardware to skip computations at a fine-grained, slice-based level rather than being forced into coarse blocks, enabling a 1.65x end-to-end speedup on H100 GPUs. VideoNSA by Enxin Song also emphasizes hardware-aware scaling, balancing token compression and salient block selection to process contexts up to 128K tokens efficiently.

Papers in this version

Showing 60 of 522 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scaling_video_transformers_via_spatiotem_f3143921,
  title = {Scaling Video Transformers via Spatiotemporal Sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-video-transformers-via-spatiotemporal-sparsity/rdr_f3143921fdcd4cf2b3ad076ad5db1c33},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_scaling_video_transformers_via_spatiotem_f3143921_v1,
  title = {Scaling Video Transformers via Spatiotemporal Sparsity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scaling-video-transformers-via-spatiotemporal-sparsity/rdr_f3143921fdcd4cf2b3ad076ad5db1c33/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}