ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4

Quentin Anthony, Ganesh Bikshandi, Praneeth Medepalli, Stephen EbertarXiv 2026 · arXiv

The move to 8-bit floating point (FP8) promised to double the throughput of deep learning workloads, but for the attention mechanism, the reality has been more complicated. While matrix multiplication gets a 2x boost, the softmax operation—which runs on vector units—remains a stubborn bottleneck.

In their paper "ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4," researchers from Zyphra introduce a suite of kernels for the AMD Instinct MI355X that tackle these hardware-specific bottlenecks through clever numerical approximations and manual assembly scheduling.

TL;DR

ZYFA provides high-performance attention kernels for AMD's CDNA4 architecture, achieving 1.16–1.76× the throughput of AMD’s default AITER kernels on real model activations. It achieves this by approximating the softmax exponential using FP8 bit-pattern properties ("Lean P") and implementing a manual "ping-pong" instruction schedule to perfectly overlap vector and matrix operations.

What Problem Does This Paper Address?

On modern GPUs like the AMD MI355X, matrix cores have evolved much faster than vector units. While FP8 matrix instructions are twice as fast as BF16, the softmax operation (which involves exponentials and divisions) still runs on the same vector pipes. This creates a "bottleneck shift": in FP8 attention, the softmax can take nearly as much time as the matrix multiplications themselves.

Furthermore, unlike NVIDIA’s Hopper architecture, the AMD CDNA4 does not have hardware-level tracking for matrix instruction completion. This makes it difficult for compilers to efficiently overlap softmax with matrix math, often leaving the expensive matrix units idle. Finally, FP8 is notoriously sensitive to numerical error in attention, where small probabilities can easily underflow to zero, degrading model accuracy.

Methods

The authors built ZYFA using three primary innovations:

  1. Lean P (The Bit-Pattern Trick): Instead of calculating a full hardware exponential for softmax, ZYFA exploits the fact that the bit pattern of an FP8 number (e4m3) effectively encodes a scaled base-2 logarithm. By using a single multiply-add and an integer conversion, they can approximate the probability directly from the score. This reduces vector cycles for softmax by over 50%.
  2. Ping-Pong Scheduling: To solve the overlap problem, the authors wrote raw assembly kernels that pair two "waves" (thread groups) on a single SIMD unit. While one wave is busy with matrix multiplications (MFMA), the other wave computes its softmax on the vector pipe.
  3. Per-Block Scaling: To combat FP8's limited range, ZYFA scales each block of 64 probabilities by a power of two before rounding. This prevents small values from underflowing to zero. Crucially, they normalize by the sum of these rounded probabilities, ensuring the attention weights always sum to exactly one.
  4. Chiplet-Aware Mapping: The MI355X consists of eight chiplets (XCDs). ZYFA remaps the thread grid so that consecutive query heads are processed on the same chiplet, maximizing L2 cache hits for the Key and Value tensors.
Fig. 2: Thread traces showing that AITER’s FP8 kernel leaves the matrix pipe idle while its waves compute exponentials and wait at barriers, and that ZYFA’s FP8 kernel keeps the matrix pipe busy most of the time. A thread trace records every instruction that a wave issues, cycle by cycle, here with the thread-trace mode of AMD’s rocprofv3 profiler. Each panel shows the two waves that share one SIMD over 2500 cycles in the middle of the loop over key blocks, where every block repeats the same work, in the FP8 kernels at dqk=128 (S=8K tokens, H=32 heads, multi-head attention). Each wave row shows the class of each instruction that the wave issues, and the matrix-pipe row below it shows when that wave’s MFMAs run. A vector instruction runs on the vector pipe during its bar in the wave row, so the wave rows already show when the vector pipe works. An MFMA keeps running on the matrix pipe for 32 to 64 cycles after its short bar in the wave row, while the wave issues other instructions, so the matrix-pipe rows show that work separately.

Datasets

The kernels were evaluated using:

  • Synthetic Benchmarks: Random Gaussian inputs (standard for peak TFLOP/s testing).
  • Captured Activations: Real $Q, K, V$ tensors extracted from Mistral-7B-v0.3, Qwen3-8B, and DeepSeek-V2-Lite during inference on the WikiText-2 dataset.
  • End-to-End Accuracy: Evaluated using the PG-19 test set to measure KL divergence and next-token prediction quality.

Key Insights and Ideas

  • The "Sustained Clock" Reality: The authors highlight that GPU clock speeds are not static; they drop as more bits flip in the hardware. Real model data (which has specific distributions) results in different power draws and clock speeds than random Gaussian noise. Benchmarking on real activations is critical for meaningful results.
  • Softmax is the New Bottleneck: In the FP8 era, the "IO-awareness" of FlashAttention is no longer enough. The arithmetic of the vector pipe is now a primary constraint that requires algorithmic changes (like Lean P) to overcome.
  • Numerical Fidelity: A major source of error in existing FP8 kernels (like AITER) is that they use unrounded probabilities for the denominator of the softmax but rounded ones for the numerator. This causes the output rows to "shrink." ZYFA's consistent use of rounded values for both significantly reduces error.

Key Results

  • Throughput: ZYFA achieves 1.16–1.76× the speed of AITER on real model activations at 16K sequence lengths. On synthetic DeepSeek-style MLA shapes, the speedup reaches 1.99×.
  • Peak Performance: At 32K tokens, ZYFA reaches 2672 TFLOP/s in FP8 (53% of datasheet peak), compared to AITER’s 39%.
  • Energy Efficiency: ZYFA delivers up to 2.05 TFLOP/J, a significant improvement over AITER's 1.47 TFLOP/J, largely due to better hardware utilization.
  • Accuracy: At head dimension 128, ZYFA’s numerical error is 62–78% lower than AITER’s assembly kernel and matches the accuracy of higher-precision Triton and Composable Kernel implementations.
Fig. 1: Throughput of ZYFA and AITER’s default kernels on the queries, keys, and values captured from one layer per model at sequence length 16K, timed at sustained clocks (the steady clock after warmup launches). L20 is layer 20. The label above each pair is AITER’s time divided by ZYFA’s time. Mistral-7B and Qwen3-8B have head dimension 128, where the FP8 kernel uses lean P, which writes each FP8 probability with one multiply-add instead of an exponential. DeepSeek- V2-Lite has the MLA shape, where the FP8 kernel uses exact P, which evaluates the exponential (Section IV-B), and AITER uses Composable Kernel (CK) in FP8 and its OPUS kernel, written with AITER’s C++ template library, in BF16. Table X gives the clock of these runs.

Limitations and Future Work

  • Scope: The current kernels only support the causal forward pass (prefill). They do not yet cover decoding, paged attention, or variable-length sequences.
  • NaN Handling: The "Lean P" approximation handles NaNs differently than standard IEEE arithmetic, which could lead to wider NaN propagation in some edge cases.
  • Future Directions: The authors suggest that further gains could be made by overlapping the transitions between query tiles and by moving the output rescaling logic to the "softmax wave" to further balance the pipeline.

The source code is available at github.com/Zyphra/zyfa.

Summary updated

Abstract

Attention is a core building block of frontier models and often dominates compute-bound prefill at long context. FP8 matrix instructions offer twice the peak throughput of BF16, but they accelerate only matrix multiplication, and the softmax between the two matrix multiplications of attention runs on vector units that FP8 does not accelerate. On the AMD Instinct MI355X, the vendor's AITER kernels reach 39% of peak FP8 matrix throughput at 32K tokens, compared with 49% in BF16. Attention is also more numerically sensitive to quantization than feed-forward layers, and unlike NVIDIA's Hopper and Blackwell GPUs, the MI355X does not track the completion of matrix instructions in hardware, so overlapping the softmax with the matrix multiplications depends on instruction order. We present ZYFA, prefill kernels for the MI355X built on three techniques: (1) computing each FP8 probability directly from its score with a multiply-add and an integer conversion instead of an exponential; (2) a ping-pong schedule in which one group of threads computes its softmax while another executes its matrix multiplications; and (3) scaling each block of probabilities up by a power of two before rounding, and normalizing by the sum of the rounded probabilities. On attention inputs captured from three models and timed at sustained clock speeds, ZYFA achieves 1.16-1.76x the throughput of AITER. On the Gaussian inputs of kernel benchmarks at 32K tokens, it achieves 1.26-1.37x at head dimension 128 and 1.99x for DeepSeek-style multi-head latent attention. The sustained clock of the MI355X depends on operand values, so throughput on random inputs does not reliably predict throughput on real model activations. At head dimension 128, ZYFA's numerical error is within a few percent of that of Composable Kernel and Triton and 62-78% lower than that of AITER's. The source code is available at https://github.com/Zyphra/zyfa.
BibTeX
@misc{anthony2026zyfa,
  author = {Quentin Anthony and Ganesh Bikshandi and Praneeth Medepalli and Stephen Ebert},
  title = {{ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4}},
  year = {2026},
  eprint = {2611.07496},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07496},
}
Figure