Fused-Softmax for efficient quantized attention

Tianxing Wu, Juuso Korhonen, Jesus Carabano BravoarXiv 2026 · arXiv

TL;DR

Fused-Softmax Attention is a training-free algorithm that breaks the "softmax bottleneck" in modern GPUs by moving vector-heavy operations onto high-speed matrix cores. By reformulating softmax into matrix-friendly operations, it achieves up to a 3.2× speedup on AMD Instinct™ MI355X GPUs while maintaining the visual quality of state-of-the-art video generation models.

What Problem Does This Paper Address?

In the world of AI hardware, not all units are created equal. Modern GPUs have seen an explosion in matrix throughput (the units that handle large multiplications), especially for low-precision formats like FP8 and FP4. However, vector throughput (the units that handle additions, exponentials, and reductions) has remained relatively stagnant.

Transformer models rely heavily on the attention mechanism, which consists of two large matrix multiplications (GEMMs) separated by a softmax operation. As we move to lower precision for the GEMMs to gain speed, the softmax—which is entirely vector work—becomes a massive bottleneck. It’s a classic case of Amdahl’s Law: even if you make the matrix parts infinitely fast, the slow vector parts will eventually dominate the total runtime. This paper aims to "hide" that vector work by turning it into matrix work.

Methods

The authors introduce Fused-Softmax Attention, which redesigns the standard FlashAttention pipeline with four key innovations:

  1. Speculative Frozen Maximum: Normally, you must find the maximum value in a row to keep the softmax numerically stable. This requires a vector pass. Instead, the authors "freeze" a speculative maximum from the first tile. If a later value exceeds this, a rollback mechanism triggers to correct the math. This allows them to move the "max subtraction" step directly into the matrix core's accumulator.
  2. Schraudolph Exponential Approximation: Calculating $e^x$ is expensive for vector units. The authors use the Schraudolph method—a bit-manipulation trick that approximates exponentials using a linear (affine) transformation. Because it's linear, this math can be "folded" into the scales and biases of the matrix multiplication itself.
  3. Two-Stage Quantization: They use a hardware-efficient path to convert scores into quantized probabilities ($P$). This reuses block reductions to both detect overflows for the rollback gate and set scales for the next operation.
  4. Row Sum as a GEMM: The denominator of the softmax (the sum of exponentials) is usually a vector reduction. The authors instead compute it as a matrix-vector product ($P$ times a column of ones), moving yet another task onto the idle matrix cores.

To keep the hardware busy, they use an eight-wave ping-pong schedule (Figure 1), where one group of waves performs matrix operations while another handles the remaining vector tasks, ensuring the execution units are never waiting on each other.

Figure 1: Eight-wave ping-pong schedule on four SIMDs. Waves 0–3 and 4–7 form phase-shifted groups: each wave overlaps its matrix operations with its SIMD partner’s softmax, and barriers preserve the phase offset across tiles.

Datasets

The researchers evaluated their kernels using:

  • Synthetic "Normal" Inputs: Standard normal distributions used to ablate components and compare against related methods like Attn-QAT and Direct-P.
  • Wan2.2-T2V-A14B: A state-of-the-art video generation model. They used 248 prompts from the VBench suite to test real-world visual quality.
  • HunyuanVideo & MiniMax-H3: Additional diffusion models used to demonstrate the generalizability of the approach.

Key Insights and Ideas

  • Matrix is the New Vector: The most profound insight is that on modern hardware, it is actually more efficient to "waste" matrix cycles to perform tasks traditionally reserved for vector units. The matrix cores are so much faster that the overhead of reformulating the math is negligible compared to the time saved on the vector units.
  • The Error Floor: A common fear with approximation is accuracy loss. The authors found that the error introduced by their Fused-Softmax is significantly lower than the "error floor" already created by quantizing the model to FP8 or FP4. In other words, the bottleneck was the problem, not the approximation.
  • Rollbacks are Rare: Speculative execution only works if you don't have to fix mistakes often. In real video generation tasks, the "rollback" gate only fired on 0.063% of tiles, meaning the speculative maximum is an incredibly safe bet.

Key Results

The performance gains on the AMD Instinct™ MI355X (CDNA 4 architecture) are substantial:

  • Raw Throughput: The kernels reached 3.0 PFLOP/s (MXFP8), 4.0 PFLOP/s (MXFP6), and 4.6 PFLOP/s (MXFP4).
  • Speedup: This represents a 2.1× to 3.2× speedup over standard BF16 attention.
  • End-to-End Impact: In the Wan2.2 video model, a hybrid schedule (using MXFP6 for sensitive steps and MXFP4 for others) delivered a 2.05× end-to-end speedup while maintaining high visual fidelity.
  • Visual Quality: VBench scores showed that the hybrid schedule was virtually indistinguishable from the high-precision BF16 baseline, whereas pure FP4 sometimes caused visible artifacts (like a cat "losing its tongue" in a generated video).
Figure 2: One SIMD pair in the Fused-Softmax schedule. While one wave issues the denominator, PV and the next QK matrix operations, its partner evaluates the row maximum, rollback gate and two-stage conversion. Prefetch and drain operations use alternating LDS slots; the rollback path below the timeline runs very rarely, only when the gate fires.

Limitations and Future Work

  • Hardware Specifics: While the mathematical principles are general, the implementation is highly tuned for the AMD CDNA 4 architecture. Porting these specific kernels to other architectures (like NVIDIA Blackwell) would require re-tuning the "ping-pong" schedules and LDS (Local Data Share) management.
  • Extreme Quantization: The authors noted that pure MXFP4 can lead to scene drifting in diffusion models. While their hybrid schedule fixes this, it suggests that even more sophisticated "precision scheduling" might be needed as we push toward 2-bit or 1-bit attention.
  • Future Directions: The authors suggest that as matrix throughput continues to outpace vector throughput, more non-matrix operations (like LayerNorm or various activation functions) will need to be "matrix-fused" to keep up.

Summary updated

Abstract

Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformulates and approximates softmax to fold its dominant operations into the attention GEMMs, moving work from vector units to matrix cores. Implemented for the AMD CDNA 4 architecture, our MXFP8, MXFP6 and MXFP4 kernels reach approximately 3.0, 4.0 and 4.6 PFLOP/s on an AMD Instinct MI355X GPU, yielding speedups of $2.1\times$, $2.8\times$ and $3.2\times$ over the BF16 attention. Importantly, the additional error introduced by Fused-Softmax remains below the error floor set by quantizing the GEMM operands. On Wan2.2-T2V-A14B, a hybrid MXFP6/4 schedule delivers up to $2.05\times$ end-to-end speedup while preserving the visual quality of the outputs. The kernels ship in AMD's open-source AITER library, and they powered AMD's MLPerf Inference v6.1 Wan2.2 submission, which ran $1.7\times$ faster than AMD's previous-round result on the same hardware. Our results show that redesigning softmax to utilize the matrix pipeline enables attention to capitalize on continued growth in low-precision matrix throughput.
BibTeX
@misc{wu2026fused,
  author = {Tianxing Wu and Juuso Korhonen and Jesus Carabano Bravo},
  title = {{Fused-Softmax for efficient quantized attention}},
  year = {2026},
  eprint = {2611.09225},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09225},
}
Figure