TL;DR
FlyWheel is a suite of high-performance attention kernels specifically engineered for Google’s Tensor Processing Units (TPUs). By redesigning data layouts and reformulating algorithms to shift work from vector units to matrix units, FlyWheel achieves up to a 4.14× speedup in softmax attention and a 1.56× boost in end-to-end LLM prefill throughput, reaching an impressive 81% of peak hardware utilization.
What Problem Does This Paper Address?
While attention mechanisms like FlashAttention have been heavily optimized for GPUs, these optimizations don't always translate well to TPU architectures. TPUs excel at massive matrix multiplications via their Matrix Units (MXUs), but they are often bottlenecked by "vector-intensive" operations like softmax reductions, rescaling, and element-wise updates.
In standard attention kernels, the MXUs frequently sit idle while the Vector Processing Units (VPUs) struggle to keep up. Furthermore, the TPU's "weight-stationary" design means that loading data into the MXU has a high overhead; if you don't reuse that data effectively, performance plummets. Existing state-of-the-art TPU kernels often achieve only 18–73% of the hardware's theoretical "roofline" throughput, leaving significant performance on the table.
Methods
The authors introduce FlyWheel, implemented in Pallas (a TPU kernel language), which employs four primary strategies:
- Coordinated Data Layouts: FlyWheel uses a "head-major" layout that packs tokens from the same head into sublanes. This maximizes operand reuse—allowing a single load of a key or value tile to serve 256 query rows—and eliminates the padding overhead seen in previous kernels like RPA-v3. Comparison of RPA-v3 and FlyWheel data layouts
- Checked Static Anchor Softmax: Instead of the standard "online softmax" which requires constant rescaling and introduces serial dependencies, FlyWheel uses a fixed reference (anchor) for exponential weights. It includes a "guarded replay" mechanism: it checks if the accumulated values remain within a safe numerical range; if they don't, it recomputes only the failing block with a safer anchor. This drastically reduces vector unit pressure.
- Hierarchical Triangular Inversion: For linear attention (specifically Gated DeltaNet), FlyWheel replaces slow, sequential vector-based forward substitution with a fast, matrix-based hierarchical inversion. This moves the heavy lifting of the triangular solve onto the MXUs. Hierarchical triangular inversion merging blocks
- Explicit Pipelining: FlyWheel explicitly interleaves independent matrix and vector operations. For example, it overlaps the score computation of one fragment with the output accumulation of the previous one, ensuring the MXU stays busy even during vector-heavy softmax steps. Pipelining across fragments to fill MXU idle gaps
Datasets and Benchmarks
The kernels were evaluated on TPU v6e (Trillium) and v7x (Ironwood). To test real-world impact, they integrated FlyWheel into the vLLM serving framework using the Qwen3.8-27B model. Evaluation benchmarks included:
- RULER: For long-context needle-in-a-haystack tasks (up to 512K tokens).
- BABILong: For reasoning over extremely long sequences.
- Video-MME: For multi-modal (video) understanding tasks.
Key Insights and Ideas
- "Matrix is Cheap, Vector is Expensive": On TPUs, it is often faster to perform more total floating-point operations if those operations can be moved from the vector units to the matrix units.
- Breaking Dependencies: The "Static Anchor" is a clever way to break the sequential chain of rescaling in online softmax. By gambling that a fixed anchor will work and only replaying on failure, FlyWheel unlocks massive parallelism.
- Layout is Destiny: Performance on TPUs is deeply tied to how data is tiled in memory. FlyWheel’s ability to handle "ragged" sequence boundaries on-chip without expensive host-side layout conversions is a major win for serving efficiency.
Key Results
- Raw Kernel Speed: FlyWheel is up to 4.14× faster than RPA-v3 and 1.80× faster than Google’s Splash Attention for softmax attention.
- Hardware Utilization: It reaches 81% of peak BF16 throughput on TPU v6e, significantly narrowing the gap to the theoretical roofline. Throughput comparison against roofline
- Linear Attention: Prefill for Gated DeltaNet is accelerated by up to 3.61×.
- End-to-End Gains: In vLLM, FlyWheel improves prefill throughput by 1.56× for 512K token contexts while maintaining the same model accuracy as baseline kernels.
Limitations and Future Work
- TPU Specificity: The optimizations are highly tailored to the TPU's VLIW and weight-stationary architecture; they are not directly portable to GPUs.
- Implementation Complexity: The "guarded replay" and hierarchical inversion require careful manual tuning and management of register pressure.
- Future Directions: The authors suggest extending these "matrix-first" principles to other non-attention operators that are currently vector-bottlenecked on TPUs. They also note that while prefill is significantly improved, decode-phase optimizations (briefly mentioned in appendices) remain an area for continued refinement.
Summary updated