Systolic Architectures for Data Reuse and Routing Efficiency

25 concepts21 papers

In modern computing, the fundamental bottleneck to performance is rarely the speed of arithmetic logic units (ALUs); it is the cost of moving data. In traditional architectures, a central memory or register file broadcasts data to multiple processing elements (PEs). This global broadcasting creates a "memory wall" and a "power wall," where the energy consumed by long-distance wiring and high signal fan-out dwarfs the energy spent on actual computation.

Systolic architectures solve this problem by replacing global data broadcasting with localized, rhythmic data circulation. Instead of returning to memory after every operation, data is read once and "pumped" through a network of locally connected PEs. Each PE performs a partial computation and passes the data directly to its neighbor in a wave-like fashion. This minimizes global wiring congestion, maximizes data reuse, and drastically reduces memory bandwidth requirements.

The Mechanics of Localized Data Flow

To understand why systolic arrays are so efficient, consider a standard matrix multiplication. In a conventional processor, computing an inner product requires repeatedly fetching elements from a register file, executing a multiply-accumulate (MAC) operation, and writing the result back.

As demonstrated by Norman P. Jouppi et al. in their analysis of the Google Tensor Processing Unit (In-Datacenter Performance Analysis of a Tensor Processing Unit), a hardware systolic array arranges PEs in a grid where data flows across rows and columns. One matrix arrives from the left, another from the top, and they intersect at the PEs. Because each intermediate result or operand is passed directly to an adjacent unit, the hardware avoids the energy tax of accessing external memory or global buses during the computation. This localized data movement is also highly effective for convolutional neural networks, where Hans Peter Graf et al. showed that passing results directly between adjacent units minimizes global data movement and reduces the overall energy footprint of the chip (A Massively Parallel Digital Learning Processor).

This principle of localized data circulation extends beyond standard dense matrices. For instance, Oscar Castañeda et al. mapped complex wireless data detection algorithms onto specialized triangular systolic arrays (Data Detection in Large Multi-Antenna Wireless Systems via Approximate Semidefinite Relaxation). By tailoring the physical layout of the PEs to match the natural data flow of triangular matrix operations, the authors eliminated memory access conflicts and minimized global communication overhead.

Topological Trade-offs: 1D vs. 2D Arrays

While a two-dimensional grid of PEs naturally maps to matrix-matrix multiplication, it introduces physical routing challenges on actual silicon, particularly on Field-Programmable Gate Arrays (FPGAs) or multi-die chiplet architectures. In a 2D grid, signals must travel both horizontally and vertically, which can lead to routing congestion and timing pressures that degrade the maximum clock frequency.

To circumvent this, researchers often collapse 2D arrays into one-dimensional (linear) systolic chains. As explored in Flexible Communication Avoiding Matrix Multiplication on FPGA with High-Level Synthesis by Johannes de Fine Licht et al., a 1D topology simplifies physical routing by bounding signal fan-out to exactly one direction. This linear connectivity is uniquely advantageous for multi-die architectures because a single line of data paths can cross the boundaries between separate silicon dies far more easily than a dense 2D mesh, maintaining stable clock frequencies as the design scales.

2D Systolic Array (High Routing Complexity)
  ↓   ↓   ↓
→ PE → PE → PE
  ↓   ↓   ↓
→ PE → PE → PE

1D Linear Systolic Chain (Simplified Routing)
→ PE → PE → PE → PE → PE

Similarly, Akshay Dua et al. utilized a highly parameterized 1D systolic array to build a runtime-flexible CNN accelerator (Systolic-CNN: An OpenCL-defined Scalable Run-time-flexible FPGA Accelerator Architecture for Accelerating Convolutional Neural Network Inference in Cloud/Edge Computing). By restricting data movement to a linear chain, the authors bounded signal fan-out and streamlined physical placement, enabling the design to achieve up to 100% utilization of the FPGA's digital signal processing (DSP) blocks without timing violations.

Hybrid and Hierarchical Variations

To maximize efficiency across diverse workloads, the strict neighbor-to-neighbor rule of systolic arrays can be modified into hybrid or hierarchical structures.

Semi-Systolic Arrays

A fully systolic array shifts all operands through the network. A semi-systolic array, by contrast, broadcasts one operand to all PEs simultaneously while shifting the other operand through the chain. Yvan Tortorella et al. demonstrated this in RedMulE, a matrix accelerator for ultra-low-power RISC-V SoCs (RedMulE: A Compact FP16 Matrix-Multiplication Accelerator for Adaptive Deep Learning on RISC-V-Based Ultra-Low-Power SoCs). This hybrid approach balances data reuse with control complexity, delivering high throughput with significantly reduced hardware overhead compared to fully systolic designs.

Hierarchical Tensor-PEs

Instead of managing data flow at the level of individual scalar values, architectures can group scalar compute units into hierarchical blocks. Zhi Gang Liu et al. introduced this concept via the Systolic Tensor Array (Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference). By organizing the array into hierarchical blocks where internal compute units share a common data-flow path, the architecture reduces the frequency of data movement between global memory and local registers, optimizing interconnect efficiency and lowering the load on the clock tree.

Matrix-Centric ISAs

To cleanly interface general-purpose processors with these specialized arrays, researchers have designed matrix-centric instruction set architectures (ISAs). As shown by Danilo Cammarata et al. with the Quadrilatero coprocessor, utilizing dedicated 2D registers allows the system to handle entire data blocks as single entities (Quadrilatero: A RISC-V programmable matrix coprocessor for low-power edge applications). This design simplifies the control logic and minimizes register file access bottlenecks, ensuring that the systolic array remains fed with data without overwhelming the host processor.

Software-Defined Systolic Arrays

The intuition behind systolic hardware—passing data directly between adjacent processing units to avoid a central memory bottleneck—is so powerful that it has been adapted into software for general-purpose processors like GPUs.

On a modern GPU, threads within a warp typically communicate by writing data to shared memory (a local scratchpad) and reading it back out. This process incurs store-load latency and consumes shared memory bandwidth. Mohamed Wahib et al. bypassed this bottleneck by implementing a Software Systolic Array Model (SSAM) (A Versatile Software Systolic Execution Model for GPU Memory-Bound Kernels).

Traditional GPU Thread Communication:
Thread 0 → Write to Shared Memory → Latency/Contention → Thread 1 Read

Software Systolic Array (Warp Shuffle):
Thread 0 ── Register-to-Register (1 Cycle) ──→ Thread 1

Instead of using shared memory, the authors leveraged hardware warp shuffle primitives to execute high-speed horizontal data exchanges directly between thread registers in a single clock cycle. This creates a software-defined systolic array where partial results flow across a virtual grid of threads. By shifting the data bottleneck from shared memory to the register file, this software model mimics the efficiency of domain-specific hardware on a fully programmable GPU.

Key Foundational Work

Fine-grained parallel RNAalifold algorithm for RNA secondary structure prediction on FPGA (2009) — contributed 1 concept elements to this direction.
A Massively Parallel Digital Learning Processor (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_systolic_array_architecture_1cc251ab,
  title = {systolic array architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/systolic-array-architecture/rdr_1cc251abb7004298afbdde2bd79c63b8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_systolic_array_architecture_1cc251ab_v1,
  title = {systolic array architecture},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/systolic-array-architecture/rdr_1cc251abb7004298afbdde2bd79c63b8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake