TL;DR
Researchers at Google have introduced Systolic Linear Attention (SLA), a redesign of gated linear attention specifically optimized for the unique architecture of TPUs. By aligning the mathematical operations of the attention layer with the TPU’s massive $256 \times 256$ systolic matrix units, SLA achieves up to a 2.5× speedup at the kernel level and trains large-scale models 11%–24% faster than existing GPU-centric implementations.
What Problem Does This Paper Address?
Modern "linear attention" models (like Gated DeltaNet or Kimi Delta Attention) were designed and tuned for NVIDIA GPUs. GPUs excel at handling small data "tiles" and performing fast vector operations. However, Google TPUs are built differently: they rely on massive Systolic Matrix Units (MXUs) that process data in large $256 \times 256$ blocks.
When you port GPU-optimized attention layers to a TPU, two major bottlenecks appear:
- Underutilization: Standard 128-wide attention heads only fill half of a TPU's 256-wide matrix tile, leaving 50% of the hardware idle.
- Vector Bottlenecks: GPU-centric models often use "per-channel decay," which requires many element-wise vector operations. On TPUs, the matrix unit is roughly 128× faster than the vector unit, meaning these operations "stall" the entire chip.
The authors argue that to get the most out of systolic accelerators, we can't just optimize the code—we have to co-design the math of the attention layer itself.
Methods
The authors propose Systolic Linear Attention (SLA), which rests on three architectural "pillars" designed to saturate the TPU's MXU:
- Wide Keys & Grouped Heads: SLA widens the key dimension to $d_k=256$ to perfectly fit the MXU tile. To keep the parameter count in check, it shares one key head across multiple value heads (Grouped Value Attention).
- Tile-Grouped Forget Gates: Instead of giving every single channel its own decay rate (which is slow on TPUs), SLA groups channels into four 64-channel tiles. This allows the decay math to be performed as a series of fast matrix multiplications on the MXU rather than slow vector operations.
- MXU-Native Chunk Inversion: Traditional "triangular solves" are sequential and run on the slow vector unit. SLA replaces this with a recursive "block-doubling" algorithm that computes the matrix inverse using only dense matrix products on the MXU.
Finally, they fuse all these operations into a single high-performance kernel that lives entirely in the TPU's on-chip memory (VMEM), eliminating the need to constantly move data back and forth to main memory (HBM).
Datasets
The researchers evaluated SLA using:
- Dolma 3.5: A massive web-scale corpus used for pre-training models.
- SQuAD & HotpotQA: Used to test the models' ability to retrieve information from long contexts (up to 64K tokens).
- RULER: A benchmark specifically designed to evaluate "needle-in-a-haystack" retrieval in long-context language models.
Key Insights and Ideas
- Hardware-Aware Math: The most important takeaway is that the "best" mathematical formulation for an algorithm depends on the hardware it runs on. A "more expressive" rule (like per-channel decay) might actually be worse if it runs so slowly that you can't train on enough data.
- The Power of Fusion: By keeping the entire attention computation on-chip in VMEM, SLA avoids the "HBM tax"—the massive time penalty of reading and writing intermediate results to main memory.
- Systolic Efficiency: The paper demonstrates that for systolic arrays, "wider is often free." Increasing the key width from 128 to 256 didn't increase the time taken for the state scan because the hardware was already paying the cost for a full 256-wide tile.
Key Results
- Kernel Speed: On TPU v7x, the SLA-R4 layer is 1.6×–2.5× faster than the fastest open-source KDA (Kimi Delta Attention) baseline.
- Training Throughput: In a large-scale study using 2,048 TPU v7x chips, SLA trained models (ranging from 6.5B to 63.1B parameters) 11%–24% faster than the baseline.
- Model Quality: Despite the "coarser" tile-grouped decay, SLA models achieved similar or slightly better perplexity (a measure of prediction accuracy) than the more complex GPU-tuned baselines.
- Memory Efficiency: SLA requires 52%–61% less temporary memory (HBM) than baselines, allowing for larger batches or longer sequences.
Limitations and Future Work
- Chunk Dependency: The current recurrence depends on the "chunk size" ($C$). While efficient for training, this adds some complexity to autoregressive decoding (inference).
- Momentum Costs: While adding momentum improved model quality, it doubles the size of the state that must be carried across devices during "context-parallel" training.
- GPU Portability: Because SLA is specifically designed for the wide tiles of TPUs, it might not provide the same benefits on GPUs with smaller tensor cores, though the authors suggest it could still be competitive.
- Future Directions: The authors suggest combining SLA's hardware-efficient mapping with even more advanced attention rules, like the decoupled erase/write gates found in GDN-2.
Summary updated