The move to 8-bit floating point (FP8) promised to double the throughput of deep learning workloads, but for the attention mechanism, the reality has been more complicated. While matrix multiplication gets a 2x boost, the softmax operation—which runs on vector units—remains a stubborn bottleneck.
In their paper "ZYFA: Speed and Accuracy of FP8 FlashAttention on AMD CDNA4," researchers from Zyphra introduce a suite of kernels for the AMD Instinct MI355X that tackle these hardware-specific bottlenecks through clever numerical approximations and manual assembly scheduling.
TL;DR
ZYFA provides high-performance attention kernels for AMD's CDNA4 architecture, achieving 1.16–1.76× the throughput of AMD’s default AITER kernels on real model activations. It achieves this by approximating the softmax exponential using FP8 bit-pattern properties ("Lean P") and implementing a manual "ping-pong" instruction schedule to perfectly overlap vector and matrix operations.
What Problem Does This Paper Address?
On modern GPUs like the AMD MI355X, matrix cores have evolved much faster than vector units. While FP8 matrix instructions are twice as fast as BF16, the softmax operation (which involves exponentials and divisions) still runs on the same vector pipes. This creates a "bottleneck shift": in FP8 attention, the softmax can take nearly as much time as the matrix multiplications themselves.
Furthermore, unlike NVIDIA’s Hopper architecture, the AMD CDNA4 does not have hardware-level tracking for matrix instruction completion. This makes it difficult for compilers to efficiently overlap softmax with matrix math, often leaving the expensive matrix units idle. Finally, FP8 is notoriously sensitive to numerical error in attention, where small probabilities can easily underflow to zero, degrading model accuracy.
Methods
The authors built ZYFA using three primary innovations:
- Lean P (The Bit-Pattern Trick): Instead of calculating a full hardware exponential for softmax, ZYFA exploits the fact that the bit pattern of an FP8 number (e4m3) effectively encodes a scaled base-2 logarithm. By using a single multiply-add and an integer conversion, they can approximate the probability directly from the score. This reduces vector cycles for softmax by over 50%.
- Ping-Pong Scheduling: To solve the overlap problem, the authors wrote raw assembly kernels that pair two "waves" (thread groups) on a single SIMD unit. While one wave is busy with matrix multiplications (MFMA), the other wave computes its softmax on the vector pipe.
- Per-Block Scaling: To combat FP8's limited range, ZYFA scales each block of 64 probabilities by a power of two before rounding. This prevents small values from underflowing to zero. Crucially, they normalize by the sum of these rounded probabilities, ensuring the attention weights always sum to exactly one.
- Chiplet-Aware Mapping: The MI355X consists of eight chiplets (XCDs). ZYFA remaps the thread grid so that consecutive query heads are processed on the same chiplet, maximizing L2 cache hits for the Key and Value tensors.
Datasets
The kernels were evaluated using:
- Synthetic Benchmarks: Random Gaussian inputs (standard for peak TFLOP/s testing).
- Captured Activations: Real $Q, K, V$ tensors extracted from Mistral-7B-v0.3, Qwen3-8B, and DeepSeek-V2-Lite during inference on the WikiText-2 dataset.
- End-to-End Accuracy: Evaluated using the PG-19 test set to measure KL divergence and next-token prediction quality.
Key Insights and Ideas
- The "Sustained Clock" Reality: The authors highlight that GPU clock speeds are not static; they drop as more bits flip in the hardware. Real model data (which has specific distributions) results in different power draws and clock speeds than random Gaussian noise. Benchmarking on real activations is critical for meaningful results.
- Softmax is the New Bottleneck: In the FP8 era, the "IO-awareness" of FlashAttention is no longer enough. The arithmetic of the vector pipe is now a primary constraint that requires algorithmic changes (like Lean P) to overcome.
- Numerical Fidelity: A major source of error in existing FP8 kernels (like AITER) is that they use unrounded probabilities for the denominator of the softmax but rounded ones for the numerator. This causes the output rows to "shrink." ZYFA's consistent use of rounded values for both significantly reduces error.
Key Results
- Throughput: ZYFA achieves 1.16–1.76× the speed of AITER on real model activations at 16K sequence lengths. On synthetic DeepSeek-style MLA shapes, the speedup reaches 1.99×.
- Peak Performance: At 32K tokens, ZYFA reaches 2672 TFLOP/s in FP8 (53% of datasheet peak), compared to AITER’s 39%.
- Energy Efficiency: ZYFA delivers up to 2.05 TFLOP/J, a significant improvement over AITER's 1.47 TFLOP/J, largely due to better hardware utilization.
- Accuracy: At head dimension 128, ZYFA’s numerical error is 62–78% lower than AITER’s assembly kernel and matches the accuracy of higher-precision Triton and Composable Kernel implementations.
Limitations and Future Work
- Scope: The current kernels only support the causal forward pass (prefill). They do not yet cover decoding, paged attention, or variable-length sequences.
- NaN Handling: The "Lean P" approximation handles NaNs differently than standard IEEE arithmetic, which could lead to wider NaN propagation in some edge cases.
- Future Directions: The authors suggest that further gains could be made by overlapping the transitions between query tiles and by moving the output rescaling logic to the "softmax wave" to further balance the pipeline.
The source code is available at github.com/Zyphra/zyfa.
Summary updated