TL;DR
QONNX introduces a standardized, high-level extension to the Open Neural Network Exchange (ONNX) format designed to represent neural networks with arbitrary bit-widths (e.g., 1-bit, 2-bit, or 4-bit). By adding just three new operators—Quant, BipolarQuant, and Trunc—it bridges the gap between quantization-aware training frameworks and hardware-specific compilers, enabling seamless deployment of ultra-efficient AI on FPGAs and ASICs.
What Problem Does This Paper Address?
Quantization is the process of reducing the numerical precision of a neural network's weights and activations to save memory and power. While standard 8-bit quantization is well-supported, researchers pushing the limits of efficiency often use "extreme" quantization, such as binary (1-bit) or ternary (2-bit) representations.
The challenge is that the current ONNX standard lacks a unified way to describe these sub-8-bit networks. Different toolkits approach the problem differently, leading to a "fragmentation" where a model trained in one framework (like Brevitas) cannot easily be ingested by a hardware compiler (like FINN or hls4ml). This paper proposes QONNX as a common language to solve this interoperability crisis.
Methods
The authors propose two main paths for representing low-precision networks:
- QCDQ (Quantize-Clip-Dequantize): A backward-compatible approach. It uses standard ONNX
QuantizeLinearandDequantizeLinearnodes but inserts aClipnode in between. This allows existing 8-bit toolchains to "see" a standard 8-bit model while effectively enforcing a narrower bit-width (like 3-bit) through clipping. - QONNX Operators: A cleaner, higher-level representation consisting of three new nodes:
Quant: Performs uniform affine quantization. Crucially, it fuses the quantization and dequantization steps, taking afloat32input and returning afloat32output. This mimics how training frameworks operate and avoids forcing a specific integer implementation too early.BipolarQuant: A specialized node for binary quantization.Trunc: Represents the truncation of least significant bits, useful for operations like quantized average pooling.
The authors also released a Python library to handle "graph cleaning." As shown in the transition from Figure 1 to Figure 2, these utilities can collapse complex, redundant graph structures generated during model export into simplified, optimized operations.
 
Datasets
The authors validated their approach using several standard benchmarks across different model architectures:
- ImageNet: Used for a 4-bit weight/4-bit activation MobileNet-V1.
- CIFAR-10: Used for VGG-like "CNV" models with various bit-widths (1-bit to 2-bit).
- MNIST: Used for tiny fully-connected (TFC) models.
These models are hosted in the newly introduced QONNX Model Zoo to encourage community sharing of low-precision architectures.
Key Insights and Ideas
- Abstraction over Implementation: By keeping the IR high-level (using
float32containers), QONNX avoids being biased toward specific byte-aligned hardware backends. The decision of how to store the bits (e.g., packed into bytes or as raw bitstreams) is left to the hardware compiler. - Leveraging Broadcast Semantics: Instead of creating separate "per-tensor" and "per-channel" quantization nodes, QONNX uses standard ONNX broadcasting. This allows for highly flexible quantization schemes, such as a model where every channel has a different bit-width.
- Relaxed Bit-Widths: The
bit_widthparameter can be a floating-point value. While hardware eventually uses integers, this allows researchers to model non-power-of-two intervals to analyze potential overflows more precisely.
Key Results
The paper demonstrates a complete end-to-end workflow where models from training libraries (QKeras, Brevitas) are converted to QONNX and then successfully compiled by FPGA toolchains (FINN, hls4ml).
Figure 5 illustrates the "Pareto frontier" of the models in the QONNX zoo, showing the trade-off between accuracy and BOPs (Bit Operations). For example:
- MobileNet-w4a4 (4-bit) achieves 71.14% accuracy on ImageNet.
- CNV-w2a2 (2-bit) achieves 89.03% accuracy on CIFAR-10.

Limitations and Future Work
- Activation Support: Currently, the FINN ingestion path for QONNX is limited to specific activations like ReLU and HardTanh. Expanding this to more general non-linearities is a priority.
- Optimization: The hls4ml toolchain does not yet treat power-of-two scales as a special case for optimization, which could further improve hardware efficiency.
- Standardization: The ultimate goal of the authors is to propose these three operators for official inclusion in the main ONNX standard, moving beyond a custom extension to a global industry standard.
Summary updated