SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, Jianfei Chen

TL;DR

SpargeAttn is a universal, training-free sparse attention mechanism that accelerates inference across text, image, and video models without sacrificing quality. By combining selective token compression with a warp-level online softmax filter, it achieves up to a 1.83x end-to-end model speedup and runs 2.5x–5x faster at the kernel level compared to existing attention baselines.

What Problem Does This Paper Address?

As sequence lengths explode in modern large models—reaching up to 128K tokens in long-context LLMs and video generation transformers—the quadratic time complexity ($O(N^2)$) of the attention mechanism becomes a massive computational bottleneck.

While attention maps are inherently sparse (meaning many values drop close to zero after the softmax operation), exploiting this sparsity universally has proven difficult. Existing sparse attention methods are typically specialized: pattern-based methods rely on fixed structures like sliding windows or attention sinks, which work well for text but fail on the highly varied attention maps found in image and video diffusion models (see Figure 2). Meanwhile, dynamic sparse methods often introduce too much prediction overhead or compress tokens too aggressively, leading to severe quality degradation.

SpargeAttn introduces a universal, training-free sparse attention operator that efficiently accelerates diverse models across multiple modalities without compromising end-to-end task metrics.

Figure 2. Some sampled patterns of attention map P in video, image, and language generation models.

Methods

SpargeAttn implements a two-stage online filtering mechanism integrated directly into an 8-bit quantized attention framework (SageAttention):

  1. Selective Token Compression (Stage 1): Instead of uniformly compressing sequences, SpargeAttn calculates the mean cosine similarity within blocks of Query ($Q$) and Key ($K$) tokens. Blocks with high self-similarity are compressed into a single representative token to quickly predict a coarse, low-overhead attention mask. Crucially, blocks with low self-similarity ("fix blocks") are always computed fully to prevent the loss of critical information.
  2. Sparse Warp Online Softmax (Stage 2): Operating at the GPU warp level within the FlashAttention inner loop, this filter compares local maximum values to global maximums on the fly. If the difference indicates that a block's softmax values will be negligibly small, the subsequent matrix multiplication with the Value ($V$) matrix is skipped entirely, incurring zero extra overhead.
  3. HilbertCurve Permutation: To maximize block self-similarity in image and video models, SpargeAttn reorders 3D visual tokens along a space-filling Hilbert curve before flattening them. This preserves spatial-temporal locality far better than standard row-major flattening.
Figure 3. Workflow of SpargeAttn.

Datasets

The authors evaluated SpargeAttn across three distinct modalities using standard benchmarks:

  • Text (Llama 3.1 8B): Evaluated on WikiText (for perplexity), LongBench, InfiniteBench (En.MC), and the Needle-in-a-Haystack task (for long-context retrieval accuracy).
  • Video (CogVideoX 2B, Mochi, Open-Sora-Plan): Evaluated using the Open-Sora prompt sets across metrics including CLIPSIM/CLIP-Temp (text-video alignment), VQA-a/VQA-t (video quality), and Flow-score (temporal consistency).
  • Image (Flux .1-dev, Stable-Diffusion 3.5 Large): Evaluated on COCO annotations using FID (fidelity), Clipscore (alignment), and ImageReward (human preference).

Key Insights and Ideas

  • Modality-Agnostic Sparsity via Self-Similarity: Rather than assuming a geometric pattern (like a sliding window), SpargeAttn dynamically adapts to any modality by analyzing token self-similarity. This allows it to bridge the gap between the structured sparsity of language models and the diffuse sparsity of vision models.
  • The Necessity of "Fix Blocks": Pure token compression methods often drop crucial blocks, degrading model accuracy. Identifying and preserving non-self-similar blocks is the key mechanism that allows SpargeAttn to achieve zero metric degradation.
  • Locality Preservation via Hilbert Curves: Standard row-major flattening destroys the spatial relationships of 2D/3D visual data. Applying a Hilbert curve reordering significantly boosts block self-similarity, unlocking higher attention sparsity for visual models (see Figure 4 in the paper).

Key Results

  • End-to-End Speedups: SpargeAttn achieves a 1.83x end-to-end speedup on the Mochi video generation model on an NVIDIA L40 GPU, reducing inference time from 1897 seconds to 1037 seconds with absolutely no video quality loss.
  • Kernel Efficiency: The attention kernel runs 2.5x to 5x faster than existing dense and sparse attention baselines (such as MInference and FlexPrefill) under varying sparsity levels.
  • Zero Quality Loss: Across all tasks—whether measuring text perplexity, image FID, or video consistency—SpargeAttn maintains or slightly improves upon full-attention baselines, whereas previous sparse methods suffered severe quality degradation on non-text modalities.
Figure 1. SpargeAttn can achieve 1.83x speedup on Mochi on L40 GPU, with no video quality loss.

Limitations and Future Work

While SpargeAttn is highly universal, its optimal performance relies on a set of three hyperparameters ($\tau, \theta, \lambda$) determined via a grid search per model layer. As shown in the paper's appendix, attention sparsity varies significantly across different layers and heads, meaning a single global hyperparameter set might underperform. Future work could explore automated, dynamic hyperparameter adaptation during inference to eliminate manual tuning.

Summary updated

Abstract

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The code is available at https://github.com/thu-ml/SpargeAttn.
BibTeX
@inproceedings{zhang2025spargeattention,
  author = {Jintao Zhang and Chendong Xiang and Haofeng Huang and Jia Wei and Haocheng Xi and Jun Zhu and Jianfei Chen},
  title = {{SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=74c3Wwk8Tc},
}
Figure