QONNX: Representing Arbitrary-Precision Quantized Neural Networks

Alessandro Pappalardo, Yaman Umuroglu, Michaela Blott, Jovan Mitrevski, Benjamin Hawks, Nhan Tran, Vladimir Loncar, Sioni Summers, Hendrik Borras, Jules Muhizi, Matthew Trahms, Shih-Chieh Hsu, Scott Hauck, Javier M. Duarte

TL;DR

This paper introduces QONNX, a new extension to the ONNX format that allows for the representation of arbitrary-precision quantized neural networks (QNNs). This enables more flexible and efficient deployment of QNNs on diverse hardware platforms by providing a high-level, yet expressive, intermediate representation.

What Problem Does This Paper Address?

Neural networks often achieve significant performance gains in terms of speed and memory usage by reducing the numerical precision of their weights and activations, a process known as quantization. However, representing these quantized neural networks (QNNs) in a standardized intermediate representation (IR) like ONNX has been challenging due to a lack of consensus on how to capture quantization information. Existing ONNX formats often lead to IR duplication, limitations in representing sub-8-bit precision, and difficulties in handling various quantization schemes (e.g., channel-wise vs. layer-wise). This paper addresses the need for a unified, flexible, and backward-compatible way to represent QNNs in ONNX, facilitating interoperability between training frameworks and hardware deployment tools.

Methods

The authors propose a multi-stage approach:

  1. Backward-Compatible Extensions: They first introduce two variants to existing ONNX quantization formats:
  2. Quantized operator format with clipping: Leverages integer clipping to represent sub-8-bit quantization within existing ONNX operators.
  3. Quantize-clip-dequantize (QCDQ) format: Extends the QDQ format by inserting a Clip operation between QuantizeLinear and DequantizeLinear to model lower precision.
  4. QONNX: A High-Level IR: They then introduce a novel, higher-level ONNX format called QONNX. This format introduces three new operators:
  5. Quant: Performs uniform affine quantization, similar to Eq. 1 in the paper.
  6. BipolarQuant: A specialized operator for binary quantization.
  7. Trunc: Represents the truncation of least significant bits (LSBs) from a quantized value, preserving the input's scale and zero-point.

    A key design choice for QONNX operators is that they fuse a dequantization operation at the output, meaning they take float32 input and produce float32 output. This abstracts away the low-level integer representation, making it easier for training frameworks to target. QONNX also leverages broadcast semantics to handle various quantization schemes (tensor-wise, channel-wise, etc.) flexibly.

  8. Software Utilities and Model Zoo: The paper also presents software utilities for working with QONNX, including execution, shape inference, and graph optimizations. Additionally, they introduce the QONNX Model Zoo to share pre-quantized QNN models.

Datasets

The paper mentions the use of standard datasets for training and evaluating the models included in the QONNX Model Zoo:

  • ImageNet
  • CIFAR-10
  • MNIST

The QONNX Model Zoo itself can be considered a benchmark collection, featuring models like MobileNet-w4a4, CNV-w1a1, CNV-w1a2, CNV-w2a2, TFC-w1a1, TFC-w1a2, and TFC-w2a2, which differ in their architecture, bit widths, and target datasets.

Key Insights and Ideas

  • Abstraction for Flexibility: The core insight is that a high-level representation like QONNX, which abstracts away the low-level integer details and fuses quantization/dequantization, is crucial for enabling broad compatibility with training frameworks and diverse hardware targets.
  • Backward Compatibility is Key: Introducing clipping-based extensions to existing ONNX formats (QCDQ) is a pragmatic step to ensure that existing toolchains can still process and benefit from lower-precision representations, even if not optimally.
  • Broadcast Semantics for Expressiveness: Using broadcast semantics for scale, zero-point, and bit-width in QONNX operators allows for a highly flexible representation of various quantization schemes without needing explicit operator variants for each.
  • Model Zoo as a Catalyst: The creation of a QONNX Model Zoo is a practical contribution that lowers the barrier to entry for researchers and developers wanting to experiment with and deploy QNNs using this new format.

Key Results

The paper demonstrates the successful integration of QONNX with:

  • QKeras: Conversion of QKeras models to QONNX is shown, preserving quantization information.
  • Brevitas: Brevitas can export models to QONNX, QCDQ, and other ONNX variants.
  • FINN and hls4ml: These FPGA compiler toolchains can ingest QONNX models, enabling hardware deployment.

The QONNX Model Zoo provides a collection of pre-quantized models with varying bit widths and architectures, allowing for comparisons of accuracy versus computational cost (BOPs), as illustrated in Figure 5.

Limitations and Future Work

  • Limitations of Clipping-Based Methods: The QCDQ and operator-with-clipping methods are limited to sub-8-bit precision and cannot represent finer-grained schemes like channel-wise bit widths due to the scalar nature of the Clip operator.
  • Future Work: The authors express a goal to propose the QONNX operators to the ONNX community for potential inclusion in the official ONNX standard. They also mention ongoing work to improve the handling of power-of-two scales in hls4ml by potentially using fractional values in ap_fixed and ac_fixed representations. Further development of the QONNX Model Zoo is also planned.

Summary updated

Abstract

We present extensions to the Open Neural Network Exchange (ONNX) intermediate representation format to represent arbitrary-precision quantized neural networks. We first introduce support for low precision quantization in existing ONNX-based quantization formats by leveraging integer clipping, resulting in two new backward-compatible variants: the quantized operator format with clipping and quantize-clip-dequantize (QCDQ) format. We then introduce a novel higher-level ONNX format called quantized ONNX (QONNX) that introduces three new operators -- Quant, BipolarQuant, and Trunc -- in order to represent uniform quantization. By keeping the QONNX IR high-level and flexible, we enable targeting a wider variety of platforms. We also present utilities for working with QONNX, as well as examples of its usage in the FINN and hls4ml toolchains. Finally, we introduce the QONNX model zoo to share low-precision quantized neural networks.
BibTeX
@misc{pappalardo2022qonnx,
  author = {Alessandro Pappalardo and Yaman Umuroglu and Michaela Blott and Jovan Mitrevski and Benjamin Hawks and Nhan Tran and Vladimir Loncar and Sioni Summers and Hendrik Borras and Jules Muhizi and Matthew Trahms and Shih-Chieh Hsu and Scott Hauck and Javier M. Duarte},
  title = {{QONNX: Representing Arbitrary-Precision Quantized Neural Networks}},
  year = {2022},
  eprint = {2206.07527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.07527},
}
Figure