MiniFloat-NN and ExSdotp: An ISA Extension and a Modular Open Hardware Unit for Low-Precision Training on RISC-V cores

Luca Bertaccini, Gianna Paulin, Tim Fischer, Stefan Mach, Luca Benini

TL;DR

This paper introduces MiniFloat-NN, a new RISC-V instruction set extension, and ExSdotp, a novel hardware unit designed to accelerate low-precision neural network (NN) training. By enabling efficient computation with 8-bit and 16-bit floating-point formats, this work significantly boosts performance and energy efficiency for NN training on RISC-V architectures.

What Problem Does This Paper Address?

The demand for neural network (NN) training is skyrocketing, requiring massive computational power and energy. While low-precision formats (like 8-bit floating-point) have shown great promise for reducing memory footprints and improving inference efficiency, their adoption for training has been slower. This is because training often demands a higher dynamic range and precision than inference, traditionally necessitating 16-bit or even 32-bit floating-point formats. Existing hardware solutions for low-precision training are often proprietary, lack flexibility, or don't efficiently handle the core operations of NN training, such as the sum of dot products, especially when accumulating results in a higher precision format. This paper aims to bridge this gap by providing an open-source, flexible, and efficient hardware and instruction set solution for low-precision NN training.

Methods

The core innovation lies in two components:

  • MiniFloat-NN ISA Extension: This is a set of new instructions for the RISC-V architecture specifically designed for low-precision NN training. It introduces support for two 8-bit and two 16-bit floating-point formats, expanding the capabilities beyond standard RISC-V floating-point extensions.
  • ExSdotp Unit: This is a novel hardware module designed to efficiently implement the key operations of the MiniFloat-NN extension. It supports "expanding" sum-of-dot-product (ExSdotp) instructions, where the result of multiplying two low-precision numbers is accumulated into a higher-precision format. This is crucial for maintaining accuracy during training. The ExSdotp unit also handles three-term additions (Vsum and ExVsum) efficiently. A key design choice is a "fused" implementation, which avoids precision losses that can occur with cascaded, non-fused operations and reduces area and critical path delays.

The authors integrated this ExSdotp unit into an existing open-source floating-point unit (FPnew) and then embedded this enhanced FPU into an open-source RISC-V core cluster (based on Snitch cores). This creates a complete, open-source platform for exploring low-precision NN training.

Datasets

The paper does not explicitly mention specific datasets used for training NNs. Instead, it focuses on evaluating the hardware's performance and efficiency using general-matrix-multiplication (GEMM) kernels, which are fundamental building blocks for many NN operations. These kernels are run with various matrix sizes and different low-precision floating-point formats (FP8, FP16, FP16alt, FP32) to assess performance and accuracy.

Key Insights and Ideas

  • Expanding Operations are Key for Training: The paper highlights that for NN training, simply using low-precision formats isn't enough. The ability to accumulate results in a higher precision (expanding operations) is critical for maintaining accuracy, especially when dealing with narrow formats like FP8.
  • Fused ExSdotp Beats Cascaded FMAs: A fused ExSdotp unit offers significant advantages over simply chaining together multiple fused multiply-add (FMA) units. It provides better accuracy by avoiding intermediate rounding, reduces hardware area by approximately 30%, and shortens the critical path.
  • RISC-V as a Flexible Platform: The use of open-source RISC-V cores and FPUs (like Snitch and FPnew) allows for rapid integration and exploration of new ISA extensions and hardware modules, democratizing research in this area.
  • SIMD for Efficiency: Implementing the ExSdotp unit in a Single Instruction, Multiple Data (SIMD) fashion allows for packing multiple low-precision operations into a single instruction, dramatically increasing throughput.

Key Results

  • Area and Timing: The ExSdotp unit is approximately 30% smaller and has a 30% shorter critical path compared to a cascade of two ExFMA modules performing a similar function.
  • Performance: The MiniFloat-NN-capable Snitch cluster achieves peak utilization of 16 FLOP/cycle for 8-to-16-bit GEMMs, doubling the performance of previous FMA-based approaches.
  • Energy Efficiency: The implemented cluster achieves an impressive 575 GFLOPS/W when computing FP8-to-FP16 GEMMs at 0.8V and 1.26 GHz. This is a significant improvement over existing solutions, offering up to 7.2x higher efficiency than a native Snitch system running double-precision GEMMs.
  • Accuracy: The ExSdotp unit demonstrates better accuracy than ExFMA implementations when accumulating products, especially for lower-precision formats like FP8, with relative errors against FP64 being consistently lower.

Limitations and Future Work

  • Limited FP Format Support: While the ExSdotp unit is highly parameterized, the current implementation focuses on specific 8-bit and 16-bit formats. The paper suggests that new formats can be rapidly defined and explored.
  • Software Emulation vs. Hardware: While previous work on 8-bit FP training relied on software emulation, this paper presents a hardware implementation, validating the feasibility and benefits. However, the full impact on diverse NN training workloads still needs extensive exploration.
  • Future Work: The authors envision extending this work to support more complex NN operations and exploring the scalability of the proposed cluster architecture for larger, more demanding AI workloads. They also suggest further investigation into the optimal trade-offs between precision, performance, and energy efficiency for various NN training tasks.

Summary updated

Abstract

Low-precision formats have recently driven major breakthroughs in neural network (NN) training and inference by reducing the memory footprint of the NN models and improving the energy efficiency of the underlying hardware architectures. Narrow integer data types have been vastly investigated for NN inference and have successfully been pushed to the extreme of ternary and binary representations. In contrast, most training-oriented platforms use at least 16-bit floating-point (FP) formats. Lower-precision data types such as 8-bit FP formats and mixed-precision techniques have only recently been explored in hardware implementations. We present MiniFloat-NN, a RISC-V instruction set architecture extension for low-precision NN training, providing support for two 8-bit and two 16-bit FP formats and expanding operations. The extension includes sum-of-dot-product instructions that accumulate the result in a larger format and three-term additions in two variations: expanding and non-expanding. We implement an ExSdotp unit to efficiently support in hardware both instruction types. The fused nature of the ExSdotp module prevents precision losses generated by the non-associativity of two consecutive FP additions while saving around 30% of the area and critical path compared to a cascade of two expanding fused multiply-add units. We replicate the ExSdotp module in a SIMD wrapper and integrate it into an open-source floating-point unit, which, coupled to an open-source RISC-V core, lays the foundation for future scalable architectures targeting low-precision and mixed-precision NN training. A cluster containing eight extended cores sharing a scratchpad memory, implemented in 12 nm FinFET technology, achieves up to 575 GFLOPS/W when computing FP8-to-FP16 GEMMs at 0.8 V, 1.26 GHz.
BibTeX
@inproceedings{bertaccini2022minifloat,
  author = {Luca Bertaccini and Gianna Paulin and Tim Fischer and Stefan Mach and Luca Benini},
  title = {{MiniFloat-NN and ExSdotp: An ISA Extension and a Modular Open Hardware Unit for Low-Precision Training on RISC-V cores}},
  booktitle = {ARITH},
  year = {2022},
}
Figure