Exploring FPGA designs for MX and beyond

Ebby Samson, Naveen Mellempudi, Wayne Luk, George A. Constantinides

TL;DR

This paper introduces the first open-source FPGA implementation of the new MX low-precision computation standard, along with a PyTorch library to facilitate its use. Their work demonstrates that MX formats, particularly narrow integer types, offer a compelling trade-off between accuracy and hardware area on FPGAs, outperforming traditional quantization methods for certain use cases.

What Problem Does This Paper Address?

The ever-increasing demand for more powerful neural networks, coupled with the need to deploy them in resource-constrained environments (like edge devices), drives the need for efficient computation. Traditional methods often rely on high-precision floating-point formats (like FP32), which consume significant computational power and memory. While low-precision quantization techniques aim to reduce this overhead, they can lead to accuracy degradation. The new Open Compute Project (OCP) MX standard was developed to enable efficient neural network implementation using very low-precision formats (fewer than 8 bits) while minimizing accuracy loss. However, realizing the full potential of these novel formats, especially on hardware like FPGAs which offer flexibility beyond standard GPUs, requires dedicated implementations and exploration tools. This paper addresses the gap by providing the first open-source FPGA implementation of the MX standard and a user-friendly software infrastructure for its exploration.

Methods

The authors present a comprehensive solution comprising two main components:

  • Open-Source FPGA IP Cores: They developed a library of hardware components that fully support the MX standard's arithmetic operations and concrete formats. Crucially, they also made implementation choices for aspects left undefined by the standard, such as handling special values (NaNs, Infs) and internal precision for accumulation. They leverage the reconfigurable nature of FPGAs to support arbitrary precision integer and floating-point formats, allowing for flexible block sizes beyond the standard's default.
  • Open-Source PyTorch Infrastructure: To enable easy exploration and integration, they created a PyTorch library that quantizes models into MX formats. This library is integrated with the existing Brevitas library, allowing for Quantization-Aware Training (QAT) and design space exploration. Their infrastructure extends Brevitas by offering more choices for scale types and computation methods, including those not explicitly defined by the MX standard.

Datasets

The primary dataset used for evaluation is ImageNet ILSVRC12. They used a pre-trained ResNet-18 model on this dataset as a starting point for their experiments.

Key Insights and Ideas

  • FPGA Advantage for Narrow Formats: FPGAs shine when implementing custom datapaths for formats like INT5 or FP6, which are not natively supported on GPUs. This flexibility allows them to exploit the smaller area footprints offered by these narrow formats.
  • MXINT Dominance for Area/Accuracy Trade-off: For low area requirements, MXINT formats (block floating-point) with coarse-grained scales offer a compelling Pareto-optimal point, achieving accuracy close to the FP32 baseline.
  • QAT is Crucial for Low-Bit Formats: Quantization-Aware Training (QAT) significantly improves the accuracy of very low-bit formats (e.g., 4-bit MXINT and MXFP), making them feasible for implementations with limited area.
  • Block Size Matters: While bit width has the most significant impact on error reduction, block size also plays a role, with smaller block sizes generally leading to lower error.
  • Flexibility Beyond the Standard: The authors' infrastructure allows exploration of scale types and computation methods beyond those defined in the MX standard, opening up new avenues for optimization.

Key Results

The paper demonstrates the usability and efficacy of their libraries by implementing example neural networks like ResNet-18 on ImageNet. Their testing shows that MX formats, especially narrow integer types (MXINT4/5/6/7), offer a significant improvement in the area/error trade-off compared to traditional per-tensor or per-channel quantization on FPGAs. For instance, QAT makes 4-bit MXINT and MXFP formats feasible, and MXINT5/6/7 formats achieve near-baseline accuracy.

Limitations and Future Work

  • FP8 Implementation Complexity: The paper notes that the FP8 concrete formats require special handling for special values (NaNs, Infs) and overflow/saturating modes, which adds complexity. Their implementation supports these, but it's an area that requires careful consideration.
  • Area of FP8 Accumulation: The authors mention that their Dot implementation's error-free accumulation scales with O(2) and uses large area for FP8 formats, leading to their omission from some plots.
  • Future Work: The authors suggest several future directions:
  • Exploring mixed-precision models with different quantization schemes (MX and non-MX).
  • Investigating different scale formats (FP/INT) and scale computation methods.
  • Creating IP cores for the best schemes identified through their exploration.
  • Further exploration of the trade-off between inference accuracy and FPGA area for various MX formats.

Summary updated

Abstract

A number of companies recently worked together to release the new Open Compute Project MX standard for low-precision computation, aimed at efficient neural network implementation. In this paper, we describe and evaluate the first open-source FPGA implementation of the arithmetic defined in the standard. Our designs fully support all the standard’s concrete formats for conversion into and out of MX formats and for the standard-defined arithmetic operations, as well as arbitrary fixed-point and floating-point formats. Certain elements of the standard are left as implementation-defined, and we present the first concrete FPGA-inspired choices for these elements, which we outline in the paper. Our library of optimized hardware components is available open source, and can be used to build larger systems. For this purpose, we also describe and release an open-source Pytorch library for quantization into the new standard, integrated with the Brevitas library so that the community can develop novel neural network designs quantized with MX formats in mind. We demonstrate the usability and efficacy of our libraries via the implementation of example neural networks such as ResNet-18 on the ImageNet ILSVRC12 dataset. Our testing shows that MX is very effective for formats such as INT5 or FP6 which are not natively supported on GPUs. This gives FPGAs an advantage as they have the flexibility to implement a custom datapath and take advantage of the smaller area footprints offered by these formats.
BibTeX
@inproceedings{samson2024exploring,
  author = {Ebby Samson and Naveen Mellempudi and Wayne Luk and George A. Constantinides},
  title = {{Exploring FPGA designs for MX and beyond}},
  booktitle = {FPL},
  year = {2024},
  url = {https://openreview.net/forum?id=3XDOzZ6kLL},
}

Loading related content...

Figure