Context Dependent Sparse Attention for Efficient Transformers

This direction is merged into another direction: adaptive attention sparsity. Its last version stays readable and citable here.

Version 1 of 113 papers18 conceptsfrozen 2026-10-04

The fundamental bottleneck of the Transformer architecture is the quadratic complexity of self-attention. As the sequence length $N$ grows, the cost of computing the relationship between every pair of tokens grows by $N^2$. To scale to long documents or high-resolution video, researchers initially turned to static sparsity: fixed geometric patterns like sliding windows or global "sink" tokens.

However, static patterns are a blunt instrument. They assume that a token’s importance is determined by its position rather than its content. Recent research has shifted toward Context-Dependent Sparse Attention, where the model dynamically decides which tokens to attend to at runtime based on the input data.

The Intuition: Content Over Geometry

In a static sparse model, a token at position $i$ might only look at tokens in the range $[i-k, i+k]$. But in a complex narrative or a high-resolution video, the most relevant information might be thousands of tokens away.

Xuanyu Zhang and colleagues argue in Adaptive Attention for Sparse-based Long-sequence Transformer that rigid patterns often overlook task-critical information. By using learnable position vectors and interpolation, their $A^2$-Former allows the connectivity pattern to be a function of the input, enabling the model to "reach out" and grab distant but relevant tokens that a sliding window would miss.

This data-driven approach is also central to the work of Huiyuan Chen and others in Denoising Self-Attentive Sequential Recommendation. Instead of relying on fixed patterns, their architecture learns to identify and "zero out" irrelevant historical dependencies, effectively denoising the attention matrix to focus only on the signals that matter for the current prediction.

Learning to Route

How does a model decide where to look without first computing the full $N^2$ matrix? One common strategy is to use a "cheap" proxy to predict the expensive attention weights.

In Smart Bird: Learnable Sparse Attention for Efficient and Effective Transformer, Chuhan Wu and the team at Microsoft Research employ a tiny, low-dimensional Transformer to identify important token interactions. This "routing" mechanism identifies the most promising pairs, allowing the high-dimensional attention heads to focus their computation only on those learned sparse interactions.

Similarly, in the realm of video generation, Peiyuan Zhang and colleagues demonstrate in Faster Video Diffusion with Trainable Sparse Attention that attention heads naturally specialize. Some heads focus on spatial details (within a frame), while others focus on temporal consistency (across frames). By making these patterns trainable and data-dependent, the model can dynamically allocate its "attention budget" to the most informative structures.

Bridging the Gap to Hardware

The primary challenge with dynamic sparsity is that GPUs hate "irregular" computation. Sparse matrices with random non-zero entries are often slower than dense matrices because they break the contiguous memory access patterns that modern hardware relies on.

To solve this, researchers are developing hardware-aware dynamic sparsity. Zhaodong Chen and the team behind Dynamic N: M Fine-grained Structured Sparse Attention Mechanism propose a method that prunes attention weights into a structured "N:M" pattern (e.g., 2 out of every 4 values are non-zero). By fusing this pruning into a custom CUDA kernel, they achieve significant speedups without the accuracy loss of static pruning.

Another approach, proposed by Tianchen Zhao in PAROAttention, is to "reorganize, don't accommodate." PAROAttention reorders tokens at runtime to cluster irregular attention patterns into dense, hardware-friendly blocks. This allows for aggressive sparsification (down to 20% density) while maintaining the throughput of dense computation.

Dynamic Sparsity in Generative Pipelines

In generative models like Video Diffusion Transformers (DiTs), the "importance" of a token isn't just dependent on the input—it changes during the generation process.

Haocheng Xi and Song Han show in Sparse VideoGen that the optimal sparsity pattern fluctuates based on the current stage of the denoising process. They implement a lightweight "online profiling" mechanism that samples a small fraction of tokens to determine whether a spatial or temporal pruning strategy is best for the current step. This allows the model to adapt its computational graph on the fly as the video evolves from noise to a clear image.

Why It Works

The success of these methods suggests that attention is naturally redundant. As Yucheng Li notes in SCBench, dynamic sparsity patterns demonstrate superior expressiveness over static ones because they can adapt to the "KV cache lifecycle."

By treating the attention mask as a learnable, context-aware component rather than a fixed architectural constraint, these models break the quadratic wall. They move from asking "How can we fit this sequence into memory?" to "Which parts of this sequence actually matter right now?" This shift is what enables architectures like HAX, developed by Zhihao Zhan and others, to combine the linear efficiency of State Space Models with the precise, context-dependent retrieval of Transformers.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_context_dependent_sparse_attention_for_e_3d6fceaa,
  title = {Context Dependent Sparse Attention for Efficient Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/context-dependent-sparse-attention-for-efficient-transformers/rdr_3d6fceaadae54dacbb4dfc129dce4c07},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_context_dependent_sparse_attention_for_e_3d6fceaa_v1,
  title = {Context Dependent Sparse Attention for Efficient Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/context-dependent-sparse-attention-for-efficient-transformers/rdr_3d6fceaadae54dacbb4dfc129dce4c07/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}