Abstract: DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source (https://github.com/ucb-bar/gemmini), full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks.
Introduction
Most Deep Neural Network (DNN) accelerators are designed and evaluated in a vacuum. Researchers often report "peak TOPS" or performance on isolated kernels without considering how the accelerator behaves when integrated into a real system. In practice, factors like System-on-Chip (SoC) resource contention, operating system (OS) overhead, and inefficient programming stacks can consume up to 77% of execution time, severely bottlenecking theoretical gains.
Existing hardware generators often focus solely on the accelerator core. They lack the full-stack integration—including virtual memory support, shared cache management, and high-level software compilers—necessary to understand how these components interact in a complex, multi-core environment. Gemmini addresses this by providing a tool that generates efficient ASIC accelerators integrated into complete RISC-V SoCs with a full software stack.
The Gemmini Framework
Gemmini generates a wide design space of efficient ASIC accelerators from a flexible architectural template. Key components include:
- Hardware Template: A two-level spatial array hierarchy. At the top level, "tiles" are connected via pipeline registers. Inside each tile, processing elements (PEs) are connected combinationally. This allows Gemmini to generate various architectures, from fully-pipelined systolic arrays to parallel vector engines.
- Full-Stack Integration: Gemmini integrates the accelerator into the Chipyard framework, providing a complete SoC environment with RISC-V CPUs (such as the out-of-order BOOM core), shared L2 caches, and standard peripherals.
- Software Ecosystem: It offers a multi-level programming interface, from high-level ONNX model mapping to low-level C/C++ APIs for fine-grained control.
- Virtual Memory Support: Unlike many academic accelerators, Gemmini includes a local TLB and hardware support for virtual-to-physical address translation, allowing it to run within a Linux environment.
Evaluation Methodology
The system was evaluated using five popular DNN benchmarks: ResNet50, AlexNet, SqueezeNet v1.1, MobileNetV2, and BERT. Performance was measured using FireSim, an FPGA-accelerated simulation platform that provides cycle-exact performance measurements for full SoCs.
System-Level Insights
Gemmini's flexibility enables the exploration of complex hardware-software interactions:
- Architectural Trade-offs: By adjusting the hierarchy of tiles and PEs, designers can trade off frequency for area. For instance, a systolic-style design achieved 2.7x higher frequency than a vector-style design but consumed 1.8x more area due to pipeline registers.
- Virtual Memory Overhead: The tiled nature of DNN workloads leads to high TLB miss rates (up to 30%). However, adding simple "filter registers" to cache the last TLB hit can reduce this penalty to near zero.
- Shared Resource Management: In multi-core systems, increasing the shared L2 cache size was found to be more effective for memory-bound layers than increasing the accelerator's private scratchpad.
- Hardware im2col: Implementing
im2coltransformations in hardware significantly reduces the computational burden on the host CPU, making overall performance less sensitive to the host CPU's architecture.
Key Results
- Performance Gains: Gemmini achieved up to a 2,670x speedup for ResNet50 inference compared to an in-order Rocket CPU.
- Competitive Efficiency: When configured with similar resources, Gemmini delivers performance comparable to state-of-the-art commercial accelerators like NVDLA.
- Physical Viability: The designs have been successfully fabricated in both TSMC 16nm and Intel 22nm FinFET processes.
- Area Breakdown: In a typical configuration, the spatial array consumes about 11% of total system area, while SRAM (scratchpads and accumulators) dominates at roughly 67%.
Conclusion
Gemmini bridges the gap between isolated hardware design and real-world system performance. While current versions focus on dense DNN operations, the framework provides a foundation for future work in sparse neural networks and heterogeneous accelerator clusters. Its open-source nature allows researchers to systematically evaluate the impact of SoC-level effects on DNN acceleration.
Summary updated