Gemmini: An Agile Systolic Array Generator Enabling Systematic Evaluations of Deep-Learning Architectures

Hasan Genc, Ameer Haj-Ali, Vighnesh Iyer, Alon Amid, Howard Mao, John Charles Wright, Colin Schmidt, Jerry Zhao, Albert J. Ou, Max Banister, Yakun Sophia Shao, Borivoje Nikolic, Ion Stoica, Krste Asanovic

Abstract

DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks.

Introduction

Designing hardware for Deep Neural Networks (DNNs) is traditionally a slow, rigid process. Most ASIC accelerators are "one-off" designs optimized for specific use cases, such as cloud-scale training or edge inference. However, the optimal hardware configuration changes drastically depending on the target workload, from massive ResNet models in data centers to lightweight MobileNets on smartphones.

Architects currently face a significant tool gap: the lack of a flexible, full-stack framework capable of generating diverse systolic array designs and evaluating them accurately within a complete system—including the CPU, memory hierarchy, and operating system. Without such a framework, it is difficult to determine how architectural choices, like memory size or dataflow, will perform under real-world constraints.

The Gemmini Generator and SoC Integration

Gemmini is a hardware generator written in the Chisel hardware description language. Rather than a fixed chip design, Gemmini provides a "recipe" for creating custom accelerators. Key architectural features include:

  • Flexible Architecture: It generates 2-D systolic arrays for matrix multiplication ($C = A \times B + D$). Users can tune parameters such as array dimensions, bitwidth (e.g., 8-bit vs. 32-bit), and pipeline depth.
  • Dataflow Support: It supports both Output-Stationary (OS) and Weight-Stationary (WS) dataflows, allowing architects to optimize for data movement and power consumption.
  • SoC Integration: Gemmini is integrated with the Rocket Chip generator, allowing it to be paired with in-order (Rocket) or out-of-order (BOOM) RISC-V cores via the RoCC interface.
Gemmini SoC Architecture showing integration with Rocket Core and memory hierarchy

Evaluation Methodology and Workloads

To evaluate the generated designs, the authors utilized a suite of modern deep learning workloads, including Convolutional Neural Networks (CNNs) like MobileNet and ResNet, and various Multi-Layer Perceptron (MLP) configurations. Performance was measured using FireSim, an FPGA-accelerated simulator that provides cycle-exact results while running a full Linux OS. Furthermore, designs were pushed through a complete VLSI flow (TSMC 16nm) to obtain realistic power and area metrics.

Key Insights and Results

Gemmini-generated accelerators achieved two to three orders of magnitude speedup for DNN inference compared to a host CPU. However, the systematic evaluation revealed several critical system-level insights:

  • CPU Bottlenecks: For lightweight models like MobileNet, the systolic array is so efficient that the system becomes CPU-bound. The host processor often struggles with non-matrix tasks like "im2col" reshaping, limiting overall speedup.
  • Dataflow and Power: Switching to a Weight-Stationary (WS) dataflow significantly reduced power consumption (to approximately 0.51x of the baseline) by eliminating the need for high-bitwidth accumulators in every processing element.
  • Scalability Trade-offs: Doubling systolic array dimensions (e.g., to 32x32) can provide up to a 4x performance boost for MLPs by increasing memory bandwidth and compute throughput, though at the cost of increased area and power.
Area and Power Trade-offs across various Gemmini and SoC configurations

Conclusion and Future Work

Gemmini demonstrates that an agile, generator-based approach can produce high-performance, silicon-proven DNN accelerators. While the current version handles standard convolutions exceptionally well, future work will focus on addressing bottlenecks in depthwise convolutions and improving software mapping through integration with domain-specific languages like TVM. By providing an open-source, full-stack environment, Gemmini enables the research community to move beyond isolated hardware blocks toward true system-level optimization.

Summary updated

Abstract

DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source*, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks. * https://github.com/ucb-bar/gemmini
BibTeX
@inproceedings{genc2021gemmini,
  author = {Hasan Genc and Ameer Haj-Ali and Vighnesh Iyer and Alon Amid and Howard Mao and John Charles Wright and Colin Schmidt and Jerry Zhao and Albert J. Ou and Max Banister and Yakun Sophia Shao and Borivoje Nikolic and Ion Stoica and Krste Asanovic},
  title = {{Gemmini: An Agile Systolic Array Generator Enabling Systematic Evaluations of Deep-Learning Architectures}},
  booktitle = {DAC},
  year = {2021},
  doi = {10.1109/dac18074.2021.9586216},
}
Figure