A Scalable RISC-V Vector Processor Enabling Efficient Multi-Precision DNN Inference

Chuanning Wang, Chao Fang, Xiao Wu, Zhongfeng Wang, Jun Lin

TL;DR

The researchers introduce SPEED, a scalable RISC-V vector processor specifically optimized for multi-precision Deep Neural Network (DNN) inference. By combining customized instructions, a reconfigurable systolic array architecture, and a flexible dataflow strategy, SPEED achieves up to 2.04× better area efficiency than the state-of-the-art open-source processor, Ara.

What Problem Does This Paper Address?

As Deep Neural Networks (DNNs) move toward the edge, researchers are increasingly using multi-precision quantization (using 4-bit, 8-bit, or 16-bit integers instead of 32-bit floats) to reduce storage and power costs. However, existing RISC-V Vector (RVV) processors face three major hurdles:

  1. Limited Precision Support: Most current designs struggle with very low-bitwidth operations (like 4-bit), which are essential for high-efficiency inference.
  2. Throughput Bottlenecks: Standard vector units often lack the raw parallel processing power needed for intensive DNN workloads.
  3. Inefficient Dataflow: Suboptimal movement of data between memory and the processor leads to "data starvation" or excessive off-chip memory access, killing energy efficiency.

Methods

The authors propose a three-pronged solution integrated into the SPEED architecture:

  • Customized Instructions: Built on the standard RVV 1.0 ISA, the authors added three specialized instructions:
  • VSACFG: Configures data precision (4 to 16 bits) and the dataflow strategy.
  • VSALD: A customized load instruction that broadcasts data to all lanes, maximizing data reuse.
  • VSAM: A dedicated arithmetic instruction for high-efficiency matrix-vector operations.
  • Hardware Architecture (The Systolic Array): Instead of standard functional units, each vector lane in SPEED contains a parameterized multi-precision systolic array unit (SAU). This unit is reconfigurable: a single 16-bit multiplier can be dynamically split into four 8-bit or sixteen 4-bit multipliers, allowing the hardware to scale its throughput based on the precision required.
  • Mixed Dataflow Mapping: The processor can switch between two strategies:
  • Feature Map-First (FF): Best for large kernels, maximizing data reuse.
  • Channel-First (CF): Best for small kernels (like 1x1 convolutions), reducing the memory footprint of partial results.

Datasets

The researchers evaluated SPEED using several industry-standard DNN benchmarks to ensure real-world applicability:

  • VGG16
  • ResNet18
  • GoogLeNet
  • SqueezeNet

Performance was measured via cycle-accurate simulations (QuestaSim) and hardware synthesis using the TSMC 28nm process node.

Key Insights and Ideas

  • The "Split" Multiplier: The most clever architectural move is the dynamic multiplier. By designing the hardware to treat a 16-bit slot as sixteen 4-bit slots, they achieve massive parallelism for low-precision tasks without increasing the physical area of the processor significantly.
  • Dataflow Matters as Much as Raw Power: The "Aha!" moment in the paper is the realization that a single dataflow strategy fails as kernel sizes change. By implementing a "Mixed Strategy" that chooses between FF and CF based on the layer type, they improved area efficiency by up to 3.53× compared to the baseline.
  • Broadcast vs. Ordered Load: Standard RISC-V loads are often rigid. By implementing a broadcast load (VSALD), the processor feeds the same input data to multiple lanes simultaneously, which is perfect for the repetitive nature of convolution operations.

Key Results

  • Peak Throughput: SPEED reached 287.41 GOPS at 4-bit precision.
  • Energy Efficiency: It achieved 1335.79 GOPS/W (4-bit), making it highly suitable for battery-powered edge devices.
  • Comparison to Ara: When compared to the pioneer open-source RVV processor, Ara, SPEED showed:
  • 2.04× better area efficiency at 16-bit precision.
  • 1.63× better area efficiency at 8-bit precision.
  • 12.78× better area efficiency when comparing SPEED's 4-bit performance to Ara's best-case 8-bit performance.

Limitations and Future Work

  • Fixed Point Focus: The paper focuses heavily on integer quantization (INT4/8/16). While this is standard for inference, the architecture's performance on floating-point tasks (often needed for training or specific scientific apps) is not discussed.
  • Memory Hierarchy: The study assumes an external memory for fetching data but doesn't deeply explore the impact of different cache sizes or high-bandwidth memory (HBM) interfaces on the overall system performance.
  • Scaling: While the architecture is "scalable," the paper primarily tests a 4-lane configuration. Future work could explore how the systolic array overhead scales as the number of lanes increases to 16 or 32.

Summary updated

Abstract

RISC-V processors encounter substantial challenges in deploying multi-precision deep neural networks (DNNs) due to their restricted precision support, constrained throughput, and suboptimal dataflow design. To tackle these challenges, a scalable RISC-V vector (RVV) processor, namely SPEED, is proposed to enable efficient multi-precision DNN inference by innovations from customized instructions, hardware architecture, and dataflow mapping. Firstly, dedicated customized RISC-V instructions are proposed based on RVV extensions, providing SPEED with fine-grained control over processing precision ranging from 4 to 16 bits. Secondly, a parameterized multi-precision systolic array unit is incorporated within the scalable module to enhance parallel processing capability and data reuse opportunities. Finally, a mixed multi-precision dataflow strategy, compatible with different convolution kernels and data precision, is proposed to effectively improve data utilization and computational efficiency. We perform synthesis of SPEED in TSMC 28nm technology. The experimental results demonstrate that SPEED achieves a peak throughput of 287.41 GOPS and an energy efficiency of 1335.79 GOPS/W at 4-bit precision condition, respectively. Moreover, when compared to the pioneer open-source vector processor Ara, SPEED provides an area efficiency improvement of 2.04× and 1.63× under 16-bit and 8-bit precision conditions, respectively, which shows SPEED’s significant potential for efficient multi-precision DNN inference.
BibTeX
@misc{wang2024scalable,
  author = {Chuanning Wang and Chao Fang and Xiao Wu and Zhongfeng Wang and Jun Lin},
  title = {{A Scalable RISC-V Vector Processor Enabling Efficient Multi-Precision DNN Inference}},
  year = {2024},
  eprint = {2401.16872},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.16872},
}
Figure