Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs

Gagik Amirkhanyan, Nuojin ChengarXiv 2026 · arXiv

TL;DR

Researchers at Google have introduced Systolic Linear Attention (SLA), a redesign of gated linear attention specifically optimized for the unique architecture of TPUs. By aligning the mathematical operations of the attention layer with the TPU’s massive $256 \times 256$ systolic matrix units, SLA achieves up to a 2.5× speedup at the kernel level and trains large-scale models 11%–24% faster than existing GPU-centric implementations.

What Problem Does This Paper Address?

Modern "linear attention" models (like Gated DeltaNet or Kimi Delta Attention) were designed and tuned for NVIDIA GPUs. GPUs excel at handling small data "tiles" and performing fast vector operations. However, Google TPUs are built differently: they rely on massive Systolic Matrix Units (MXUs) that process data in large $256 \times 256$ blocks.

When you port GPU-optimized attention layers to a TPU, two major bottlenecks appear:

  1. Underutilization: Standard 128-wide attention heads only fill half of a TPU's 256-wide matrix tile, leaving 50% of the hardware idle.
  2. Vector Bottlenecks: GPU-centric models often use "per-channel decay," which requires many element-wise vector operations. On TPUs, the matrix unit is roughly 128× faster than the vector unit, meaning these operations "stall" the entire chip.

The authors argue that to get the most out of systolic accelerators, we can't just optimize the code—we have to co-design the math of the attention layer itself.

Figure 1: Overview of SLA. (a) Direct ports of GPU-tuned gated-delta layers (GDN [68], KDA [71], GDN-2 [25]) run as an unfused pipeline on TPUs: intermediates round-trip through HBM, per- channel decay rescaling in KDA and GDN-2 and in-block triangular substitution run on the VPU, and 128-wide states leave at least half of each 256 × 256 MXU tile padded. (b) SLA fuses stages 2–5 into one VMEM kernel. (c) A head with key and value widths dk = dv = 128 fills a quarter of an MXU tile, while SLA’s wide packed state fills whole tiles. (d) Decay gates are grouped into four 64-channel tiles, so intra-chunk scores are four 64-wide MXU matmuls with no VPU rescaling. (e) The chunk solve uses only block matrix products on the MXU.

Methods

The authors propose Systolic Linear Attention (SLA), which rests on three architectural "pillars" designed to saturate the TPU's MXU:

  1. Wide Keys & Grouped Heads: SLA widens the key dimension to $d_k=256$ to perfectly fit the MXU tile. To keep the parameter count in check, it shares one key head across multiple value heads (Grouped Value Attention).
  2. Tile-Grouped Forget Gates: Instead of giving every single channel its own decay rate (which is slow on TPUs), SLA groups channels into four 64-channel tiles. This allows the decay math to be performed as a series of fast matrix multiplications on the MXU rather than slow vector operations.
  3. MXU-Native Chunk Inversion: Traditional "triangular solves" are sequential and run on the slow vector unit. SLA replaces this with a recursive "block-doubling" algorithm that computes the matrix inverse using only dense matrix products on the MXU.

Finally, they fuse all these operations into a single high-performance kernel that lives entirely in the TPU's on-chip memory (VMEM), eliminating the need to constantly move data back and forth to main memory (HBM).

Datasets

The researchers evaluated SLA using:

  • Dolma 3.5: A massive web-scale corpus used for pre-training models.
  • SQuAD & HotpotQA: Used to test the models' ability to retrieve information from long contexts (up to 64K tokens).
  • RULER: A benchmark specifically designed to evaluate "needle-in-a-haystack" retrieval in long-context language models.

Key Insights and Ideas

  • Hardware-Aware Math: The most important takeaway is that the "best" mathematical formulation for an algorithm depends on the hardware it runs on. A "more expressive" rule (like per-channel decay) might actually be worse if it runs so slowly that you can't train on enough data.
  • The Power of Fusion: By keeping the entire attention computation on-chip in VMEM, SLA avoids the "HBM tax"—the massive time penalty of reading and writing intermediate results to main memory.
  • Systolic Efficiency: The paper demonstrates that for systolic arrays, "wider is often free." Increasing the key width from 128 to 256 didn't increase the time taken for the state scan because the hardware was already paying the cost for a full 256-wide tile.
Figure 2: Sequential state-scan cost on TPU v7x. A 64-step scan of (C, 256) × (256, 256) matmuls takes nearly constant time for C ∈{32, . . . , 256}, so a fixed per-step overhead, not matmul FLOPs, dominates the scan.

Key Results

  • Kernel Speed: On TPU v7x, the SLA-R4 layer is 1.6×–2.5× faster than the fastest open-source KDA (Kimi Delta Attention) baseline.
  • Training Throughput: In a large-scale study using 2,048 TPU v7x chips, SLA trained models (ranging from 6.5B to 63.1B parameters) 11%–24% faster than the baseline.
  • Model Quality: Despite the "coarser" tile-grouped decay, SLA models achieved similar or slightly better perplexity (a measure of prediction accuracy) than the more complex GPU-tuned baselines.
  • Memory Efficiency: SLA requires 52%–61% less temporary memory (HBM) than baselines, allowing for larger batches or longer sequences.

Limitations and Future Work

  • Chunk Dependency: The current recurrence depends on the "chunk size" ($C$). While efficient for training, this adds some complexity to autoregressive decoding (inference).
  • Momentum Costs: While adding momentum improved model quality, it doubles the size of the state that must be carried across devices during "context-parallel" training.
  • GPU Portability: Because SLA is specifically designed for the wide tiles of TPUs, it might not provide the same benefits on GPUs with smaller tensor cores, though the authors suggest it could still be competitive.
  • Future Directions: The authors suggest combining SLA's hardware-efficient mapping with even more advanced attention rules, like the decoupled erase/write gates found in GDN-2.

Summary updated

Abstract

Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make per-channel decay rescaling and sequential in-block solves cheap. On Google TPUs, which concentrate compute in wide $256 \times 256$ systolic matrix units backed by a slower vector unit, porting these layers directly leaves half of each matrix tile empty and bottlenecks on vector operations. We present \textbf{Systolic Linear Attention (SLA)}, which reshapes gated delta attention around the matrix unit by widening keys to $256$ with grouped value heads, grouping decay gates into four $64$-channel tiles, and computing the chunk solve entirely with block matrix products in a single fused kernel. On TPU v7x, a full \texttt{SLA-R4} layer runs $1.6\times$--$2.5\times$ faster than the open-source Tokamax KDA kernel ($1.4\times$--$2.4\times$ with exact chunk inversion), driven primarily by head sharing and kernel fusion. Across a $2{,}048$-chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models from $6.5\text{B}$ to $63.1\text{B}$ parameters, SLA trains $11\%$--$24\%$ faster than KDA at similar perplexity, and reaches slightly lower perplexity than KDA in smaller matched-seed runs.
BibTeX
@misc{amirkhanyan2026systolic,
  author = {Gagik Amirkhanyan and Nuojin Cheng},
  title = {{Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs}},
  year = {2026},
  eprint = {2611.06828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06828},
}
Figure