Abstract
Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that a missing principle is making attention algorithms IO-aware -- accounting for reads and writes between levels of GPU memory. We propose FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. We analyze the IO complexity of FlashAttention, showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes. We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method. FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3x speedup on GPT-2 (seq. length 1K), and 2.4x speedup on long-range arena (seq. length 1K-4K). FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1% accuracy).
Introduction
The "Standard Attention" mechanism used in Transformers has a fundamental scaling problem: its time and memory complexity are quadratic ($O(N^2)$) relative to the sequence length ($N$). As we try to process longer documents or higher-resolution images, the memory requirements explode, and the computation slows to a crawl.
While many "approximate" attention methods have been proposed to solve this, they often trade off model quality for speed. Crucially, these methods focus on reducing the number of floating-point operations (FLOPs) but ignore the real bottleneck on modern GPUs: Memory IO. Moving data between slow High Bandwidth Memory (HBM) and fast on-chip SRAM often takes much longer than the actual math. Standard attention is "memory-bound," meaning the GPU spends most of its time waiting for data to arrive rather than calculating.
Methods: IO-Awareness through Tiling and Recomputation
FlashAttention introduces IO-awareness to the attention mechanism, aiming to minimize the number of reads and writes to the slow HBM. The algorithm employs two primary strategies:
- Tiling: Instead of calculating the massive $N \times N$ attention matrix all at once, the algorithm breaks the input into blocks. It loads these blocks into SRAM, performs the necessary computations (including a clever incremental softmax), and then moves to the next block.
- Recomputation: In the backward pass, standard attention stores the large $N \times N$ matrix to calculate gradients. FlashAttention avoids this massive memory footprint by not storing the matrix. Instead, it stores a small normalization factor from the forward pass and recomputes the attention blocks on the fly during the backward pass.
Even though recomputation requires more total FLOPs, it significantly reduces HBM accesses, resulting in a much faster "wall-clock" runtime. The entire process is implemented in a single fused CUDA kernel to maintain fine-grained control over memory.
Results
The authors evaluated FlashAttention across a variety of standard and challenging benchmarks, including BERT-large, GPT-2, and the Long-Range Arena (LRA).
FlashAttention is up to 7.6x faster than the standard PyTorch attention implementation. It achieved a 15% end-to-end wall-clock speedup on BERT-large and a 3x speedup on GPT-2. By avoiding the materialization of the $N \times N$ matrix, memory usage drops from quadratic to linear ($O(N)$), allowing for a 20x reduction in memory footprint for long sequences.
Beyond speed, FlashAttention enables new capabilities. It allowed the first Transformer to achieve better-than-chance performance on the Path-X (16K length) and Path-256 (64K length) challenges. Furthermore, GPT-2 models using FlashAttention with a 4K context window outperformed standard GPT-2 models with a 1K window, showing that longer context directly translates to better model quality (0.7 better perplexity).
Limitations and Future Work
FlashAttention requires writing custom CUDA kernels, which is significantly more difficult than writing standard PyTorch code. This makes it harder to extend to new hardware (like TPUs) or specialized attention variants. Additionally, the current implementation is highly optimized for NVIDIA Ampere and Turing GPUs; performance gains may vary on different architectures. The authors suggest that the principle of IO-awareness should be applied to other parts of the deep learning stack beyond just attention, such as MLP layers and multi-GPU communication.
Summary updated