SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference

Jintao Zhang, Chendong Xiang, Haofeng Huang, Jia Wei, Haocheng Xi, Jun Zhu, Jianfei Chen

Abstract

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics.

Introduction

The "attention bottleneck" is a significant hurdle in scaling AI models. As context windows expand to 128K tokens and visual generation moves toward higher resolutions, the quadratic complexity of standard attention becomes prohibitively expensive. While many sparse attention methods exist to skip near-zero values, they often suffer from a lack of universality, high prediction overhead, or accuracy trade-offs that lead to degraded outputs.

SpargeAttn is a universal, training-free sparse attention mechanism designed to accelerate inference across diverse architectures, including Large Language Models (LLMs), image generators, and video diffusion models. By employing a two-stage online filter and a novel token reordering strategy, it achieves significant speedups without sacrificing model fidelity.

Methodology

SpargeAttn introduces a two-stage online filtering system integrated with the quantized SageAttention framework:

1. Selective Token Compression (Stage 1)

To predict the sparse mask efficiently, the algorithm compresses blocks of Queries ($Q$) and Keys ($K$) into single representative tokens. Crucially, this is only applied to "selective blocks" where tokens exhibit high similarity. If a block has low self-similarity ("fix blocks"), it is computed in full to ensure no critical information is lost, preventing the accuracy drops common in aggressive pruning.

Sparse Block Online Prediction mechanism for Stage 1 filtering

2. Sparse Warp Online Softmax (Stage 2)

Within the computed blocks, SpargeAttn uses a GPU warp-level filter to skip $P \cdot V$ (Probability $\times$ Value) products if the difference between local and global maximum values in the softmax is large enough to render the result insignificant. This stage incurs no extra overhead.

3. HilbertCurve Permutation

For visual models, standard row-major flattening destroys spatial locality. SpargeAttn uses a Hilbert Curve to reorder tokens, ensuring that adjacent pixels in 3D space remain close in the 1D sequence. This significantly increases block similarity and the potential for effective sparsity.

Experimental Evaluation

The researchers evaluated SpargeAttn across a wide array of benchmarks, including Llama 3.1 (8B) for language, CogVideoX and Mochi for video, and Flux.1-dev for image generation.

Key Results

  • Speed: SpargeAttn achieved a 1.83x end-to-end speedup on the Mochi video model using an L40 GPU. Compared to other sparse baselines like MInference, it was often 2.5x to 5x faster.
  • Accuracy: In the 128K context Needle-in-a-Haystack test, SpargeAttn maintained a 0.909 score, virtually identical to full attention (0.907).
  • Visual Fidelity: Qualitative results for Flux and CogVideoX show no visible degradation or artifacts, maintaining high aesthetic quality and temporal consistency.
End-to-end speedup and visual quality comparison on video generation

Insights and Limitations

A key finding is that sparsity scales with sequence length; for Llama 3.1, sparsity increased from 6.8% at 8K tokens to 54% at 128K tokens. This makes SpargeAttn most efficient when it is needed most.

While the method is training-free, it currently requires setting three hyperparameters per model layer. Automating this grid-search process and exploring performance on non-GPU hardware accelerators remain areas for future work. Overall, SpargeAttn provides a robust, plug-and-play operator for efficient, high-fidelity model inference.

Summary updated

Abstract

An efficient attention implementation is essential for large models due to its quadratic time complexity. Fortunately, attention commonly exhibits sparsity, i.e., many values in the attention map are near zero, allowing for the omission of corresponding computations. Many studies have utilized the sparse pattern to accelerate attention. However, most existing works focus on optimizing attention within specific models by exploiting certain sparse patterns of the attention map. A universal sparse attention that guarantees both the speedup and end-to-end performance of diverse models remains elusive. In this paper, we propose SpargeAttn, a universal sparse and quantized attention for any model. Our method uses a two-stage online filter: in the first stage, we rapidly and accurately predict the attention map, enabling the skip of some matrix multiplications in attention. In the second stage, we design an online softmax-aware filter that incurs no extra overhead and further skips some matrix multiplications. Experiments show that our method significantly accelerates diverse models, including language, image, and video generation, without sacrificing end-to-end metrics. The codes are available at https://github.com/thu-ml/SpargeAttn.
BibTeX
@inproceedings{zhang2025spargeattn,
  author = {Jintao Zhang and Chendong Xiang and Haofeng Huang and Jia Wei and Haocheng Xi and Jun Zhu and Jianfei Chen},
  title = {{SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=74c3Wwk8Tc},
}
Figure