QONNX: Representing Arbitrary-Precision Quantized Neural Networks

Alessandro Pappalardo, Yaman Umuroglu, Michaela Blott, Jovan Mitrevski, Ben Hawks, Nhan Tran, Vladimir Loncar, Sioni Summers, Hendrik Borras, Jules Muhizi, Matthew Trahms, Shih-Chieh Hsu, Scott Hauck, Javier Duarte

TL;DR

QONNX introduces a standardized, high-level extension to the Open Neural Network Exchange (ONNX) format designed to represent neural networks with arbitrary bit-widths (e.g., 1-bit, 2-bit, or 4-bit). By adding just three new operators—Quant, BipolarQuant, and Trunc—it bridges the gap between quantization-aware training frameworks and hardware-specific compilers, enabling seamless deployment of ultra-efficient AI on FPGAs and ASICs.

What Problem Does This Paper Address?

Quantization is the process of reducing the numerical precision of a neural network's weights and activations to save memory and power. While standard 8-bit quantization is well-supported, researchers pushing the limits of efficiency often use "extreme" quantization, such as binary (1-bit) or ternary (2-bit) representations.

The challenge is that the current ONNX standard lacks a unified way to describe these sub-8-bit networks. Different toolkits approach the problem differently, leading to a "fragmentation" where a model trained in one framework (like Brevitas) cannot easily be ingested by a hardware compiler (like FINN or hls4ml). This paper proposes QONNX as a common language to solve this interoperability crisis.

Methods

The authors propose two main paths for representing low-precision networks:

  1. QCDQ (Quantize-Clip-Dequantize): A backward-compatible approach. It uses standard ONNX QuantizeLinear and DequantizeLinear nodes but inserts a Clip node in between. This allows existing 8-bit toolchains to "see" a standard 8-bit model while effectively enforcing a narrower bit-width (like 3-bit) through clipping.
  2. QONNX Operators: A cleaner, higher-level representation consisting of three new nodes:
  3. Quant: Performs uniform affine quantization. Crucially, it fuses the quantization and dequantization steps, taking a float32 input and returning a float32 output. This mimics how training frameworks operate and avoids forcing a specific integer implementation too early.
  4. BipolarQuant: A specialized node for binary quantization.
  5. Trunc: Represents the truncation of least significant bits, useful for operations like quantized average pooling.

The authors also released a Python library to handle "graph cleaning." As shown in the transition from Figure 1 to Figure 2, these utilities can collapse complex, redundant graph structures generated during model export into simplified, optimized operations.

![Fig.

  1. Initial model export before processing.](/figures/art_e2f5349f3aa9429ea485da8fbba296c3) ![Fig.

  2. The same model after cleaning and optimization.](/figures/art_81b970223e9a4bb9a6f8dfc2c0c36ec6)

Datasets

The authors validated their approach using several standard benchmarks across different model architectures:

  • ImageNet: Used for a 4-bit weight/4-bit activation MobileNet-V1.
  • CIFAR-10: Used for VGG-like "CNV" models with various bit-widths (1-bit to 2-bit).
  • MNIST: Used for tiny fully-connected (TFC) models.

These models are hosted in the newly introduced QONNX Model Zoo to encourage community sharing of low-precision architectures.

Key Insights and Ideas

  • Abstraction over Implementation: By keeping the IR high-level (using float32 containers), QONNX avoids being biased toward specific byte-aligned hardware backends. The decision of how to store the bits (e.g., packed into bytes or as raw bitstreams) is left to the hardware compiler.
  • Leveraging Broadcast Semantics: Instead of creating separate "per-tensor" and "per-channel" quantization nodes, QONNX uses standard ONNX broadcasting. This allows for highly flexible quantization schemes, such as a model where every channel has a different bit-width.
  • Relaxed Bit-Widths: The bit_width parameter can be a floating-point value. While hardware eventually uses integers, this allows researchers to model non-power-of-two intervals to analyze potential overflows more precisely.

Key Results

The paper demonstrates a complete end-to-end workflow where models from training libraries (QKeras, Brevitas) are converted to QONNX and then successfully compiled by FPGA toolchains (FINN, hls4ml).

Figure 5 illustrates the "Pareto frontier" of the models in the QONNX zoo, showing the trade-off between accuracy and BOPs (Bit Operations). For example:

  • MobileNet-w4a4 (4-bit) achieves 71.14% accuracy on ImageNet.
  • CNV-w2a2 (2-bit) achieves 89.03% accuracy on CIFAR-10.

![Fig.

  1. Accuracy vs. Bit Operations (BOPs) for the QONNX Model Zoo.](/figures/art_3bdb379919a74f1db69344002fbbd2f2)

Limitations and Future Work

  • Activation Support: Currently, the FINN ingestion path for QONNX is limited to specific activations like ReLU and HardTanh. Expanding this to more general non-linearities is a priority.
  • Optimization: The hls4ml toolchain does not yet treat power-of-two scales as a special case for optimization, which could further improve hardware efficiency.
  • Standardization: The ultimate goal of the authors is to propose these three operators for official inclusion in the main ONNX standard, moving beyond a custom extension to a global industry standard.

Summary updated

Abstract

We present extensions to the Open Neural Network Exchange (ONNX) intermediate representation format to represent arbitrary-precision quantized neural networks. We first introduce support for low precision quantization in existing ONNX-based quantization formats by leveraging integer clipping, resulting in two new backward-compatible variants: the quantized operator format with clipping and quantize-clip-dequantize (QCDQ) format. We then introduce a novel higher-level ONNX format called quantized ONNX (QONNX) that introduces three new operators -- Quant, BipolarQuant, and Trunc -- in order to represent uniform quantization. By keeping the QONNX IR high-level and flexible, we enable targeting a wider variety of platforms. We also present utilities for working with QONNX, as well as examples of its usage in the FINN and hls4ml toolchains. Finally, we introduce the QONNX model zoo to share low-precision quantized neural networks.
BibTeX
@misc{pappalardo2022qonnx,
  author = {Alessandro Pappalardo and Yaman Umuroglu and Michaela Blott and Jovan Mitrevski and Ben Hawks and Nhan Tran and Vladimir Loncar and Sioni Summers and Hendrik Borras and Jules Muhizi and Matthew Trahms and Shih-Chieh Hsu and Scott Hauck and Javier Duarte},
  title = {{QONNX: Representing Arbitrary-Precision Quantized Neural Networks}},
  year = {2022},
  eprint = {2206.07527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.07527},
}
Figure