Abstract
DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks.
Introduction
Designing hardware for Deep Neural Networks (DNNs) is traditionally a slow, rigid process. Most ASIC accelerators are "one-off" designs optimized for specific use cases, such as cloud-scale training or edge inference. However, the optimal hardware configuration changes drastically depending on the target workload, from massive ResNet models in data centers to lightweight MobileNets on smartphones.
Architects currently face a significant tool gap: the lack of a flexible, full-stack framework capable of generating diverse systolic array designs and evaluating them accurately within a complete system—including the CPU, memory hierarchy, and operating system. Without such a framework, it is difficult to determine how architectural choices, like memory size or dataflow, will perform under real-world constraints.
The Gemmini Generator and SoC Integration
Gemmini is a hardware generator written in the Chisel hardware description language. Rather than a fixed chip design, Gemmini provides a "recipe" for creating custom accelerators. Key architectural features include:
- Flexible Architecture: It generates 2-D systolic arrays for matrix multiplication ($C = A \times B + D$). Users can tune parameters such as array dimensions, bitwidth (e.g., 8-bit vs. 32-bit), and pipeline depth.
- Dataflow Support: It supports both Output-Stationary (OS) and Weight-Stationary (WS) dataflows, allowing architects to optimize for data movement and power consumption.
- SoC Integration: Gemmini is integrated with the Rocket Chip generator, allowing it to be paired with in-order (Rocket) or out-of-order (BOOM) RISC-V cores via the RoCC interface.
Evaluation Methodology and Workloads
To evaluate the generated designs, the authors utilized a suite of modern deep learning workloads, including Convolutional Neural Networks (CNNs) like MobileNet and ResNet, and various Multi-Layer Perceptron (MLP) configurations. Performance was measured using FireSim, an FPGA-accelerated simulator that provides cycle-exact results while running a full Linux OS. Furthermore, designs were pushed through a complete VLSI flow (TSMC 16nm) to obtain realistic power and area metrics.
Key Insights and Results
Gemmini-generated accelerators achieved two to three orders of magnitude speedup for DNN inference compared to a host CPU. However, the systematic evaluation revealed several critical system-level insights:
- CPU Bottlenecks: For lightweight models like MobileNet, the systolic array is so efficient that the system becomes CPU-bound. The host processor often struggles with non-matrix tasks like "im2col" reshaping, limiting overall speedup.
- Dataflow and Power: Switching to a Weight-Stationary (WS) dataflow significantly reduced power consumption (to approximately 0.51x of the baseline) by eliminating the need for high-bitwidth accumulators in every processing element.
- Scalability Trade-offs: Doubling systolic array dimensions (e.g., to 32x32) can provide up to a 4x performance boost for MLPs by increasing memory bandwidth and compute throughput, though at the cost of increased area and power.
Conclusion and Future Work
Gemmini demonstrates that an agile, generator-based approach can produce high-performance, silicon-proven DNN accelerators. While the current version handles standard convolutions exceptionally well, future work will focus on addressing bottlenecks in depthwise convolutions and improving software mapping through integration with domain-specific languages like TVM. By providing an open-source, full-stack environment, Gemmini enables the research community to move beyond isolated hardware blocks toward true system-level optimization.
Summary updated