TL;DR
Fused-Softmax Attention is a training-free algorithm that breaks the "softmax bottleneck" in modern GPUs by moving vector-heavy operations onto high-speed matrix cores. By reformulating softmax into matrix-friendly operations, it achieves up to a 3.2× speedup on AMD Instinct™ MI355X GPUs while maintaining the visual quality of state-of-the-art video generation models.
What Problem Does This Paper Address?
In the world of AI hardware, not all units are created equal. Modern GPUs have seen an explosion in matrix throughput (the units that handle large multiplications), especially for low-precision formats like FP8 and FP4. However, vector throughput (the units that handle additions, exponentials, and reductions) has remained relatively stagnant.
Transformer models rely heavily on the attention mechanism, which consists of two large matrix multiplications (GEMMs) separated by a softmax operation. As we move to lower precision for the GEMMs to gain speed, the softmax—which is entirely vector work—becomes a massive bottleneck. It’s a classic case of Amdahl’s Law: even if you make the matrix parts infinitely fast, the slow vector parts will eventually dominate the total runtime. This paper aims to "hide" that vector work by turning it into matrix work.
Methods
The authors introduce Fused-Softmax Attention, which redesigns the standard FlashAttention pipeline with four key innovations:
- Speculative Frozen Maximum: Normally, you must find the maximum value in a row to keep the softmax numerically stable. This requires a vector pass. Instead, the authors "freeze" a speculative maximum from the first tile. If a later value exceeds this, a rollback mechanism triggers to correct the math. This allows them to move the "max subtraction" step directly into the matrix core's accumulator.
- Schraudolph Exponential Approximation: Calculating $e^x$ is expensive for vector units. The authors use the Schraudolph method—a bit-manipulation trick that approximates exponentials using a linear (affine) transformation. Because it's linear, this math can be "folded" into the scales and biases of the matrix multiplication itself.
- Two-Stage Quantization: They use a hardware-efficient path to convert scores into quantized probabilities ($P$). This reuses block reductions to both detect overflows for the rollback gate and set scales for the next operation.
- Row Sum as a GEMM: The denominator of the softmax (the sum of exponentials) is usually a vector reduction. The authors instead compute it as a matrix-vector product ($P$ times a column of ones), moving yet another task onto the idle matrix cores.
To keep the hardware busy, they use an eight-wave ping-pong schedule (Figure 1), where one group of waves performs matrix operations while another handles the remaining vector tasks, ensuring the execution units are never waiting on each other.
Datasets
The researchers evaluated their kernels using:
- Synthetic "Normal" Inputs: Standard normal distributions used to ablate components and compare against related methods like Attn-QAT and Direct-P.
- Wan2.2-T2V-A14B: A state-of-the-art video generation model. They used 248 prompts from the VBench suite to test real-world visual quality.
- HunyuanVideo & MiniMax-H3: Additional diffusion models used to demonstrate the generalizability of the approach.
Key Insights and Ideas
- Matrix is the New Vector: The most profound insight is that on modern hardware, it is actually more efficient to "waste" matrix cycles to perform tasks traditionally reserved for vector units. The matrix cores are so much faster that the overhead of reformulating the math is negligible compared to the time saved on the vector units.
- The Error Floor: A common fear with approximation is accuracy loss. The authors found that the error introduced by their Fused-Softmax is significantly lower than the "error floor" already created by quantizing the model to FP8 or FP4. In other words, the bottleneck was the problem, not the approximation.
- Rollbacks are Rare: Speculative execution only works if you don't have to fix mistakes often. In real video generation tasks, the "rollback" gate only fired on 0.063% of tiles, meaning the speculative maximum is an incredibly safe bet.
Key Results
The performance gains on the AMD Instinct™ MI355X (CDNA 4 architecture) are substantial:
- Raw Throughput: The kernels reached 3.0 PFLOP/s (MXFP8), 4.0 PFLOP/s (MXFP6), and 4.6 PFLOP/s (MXFP4).
- Speedup: This represents a 2.1× to 3.2× speedup over standard BF16 attention.
- End-to-End Impact: In the Wan2.2 video model, a hybrid schedule (using MXFP6 for sensitive steps and MXFP4 for others) delivered a 2.05× end-to-end speedup while maintaining high visual fidelity.
- Visual Quality: VBench scores showed that the hybrid schedule was virtually indistinguishable from the high-precision BF16 baseline, whereas pure FP4 sometimes caused visible artifacts (like a cat "losing its tongue" in a generated video).
Limitations and Future Work
- Hardware Specifics: While the mathematical principles are general, the implementation is highly tuned for the AMD CDNA 4 architecture. Porting these specific kernels to other architectures (like NVIDIA Blackwell) would require re-tuning the "ping-pong" schedules and LDS (Local Data Share) management.
- Extreme Quantization: The authors noted that pure MXFP4 can lead to scene drifting in diffusion models. While their hybrid schedule fixes this, it suggests that even more sophisticated "precision scheduling" might be needed as we push toward 2-bit or 1-bit attention.
- Future Directions: The authors suggest that as matrix throughput continues to outpace vector throughput, more non-matrix operations (like LayerNorm or various activation functions) will need to be "matrix-fused" to keep up.
Summary updated