Abstract
Model quantization is a critical technique for enabling efficient inference in Deep Neural Networks (DNNs) by reducing the bit-width of model parameters. However, existing methods often struggle with either inefficient data encoding, leading to poor compression, or require computationally expensive quantization-aware training (QAT). This work introduces Adaptive Floating-Point (AFP), a novel variant of the standard IEEE-754 floating-point format that features flexible configurations for exponent and mantissa segments. By leveraging AFP for post-training quantization, we significantly enhance model compression rates without requiring retraining or sacrificing accuracy. Furthermore, AFP eliminates the need for the intensive de-quantization steps common in dynamic quantization frameworks. Our experiments on ResNet-50 and MobileNet-v2 demonstrate near-lossless accuracy (0.04% to 0.6% degradation) while reducing energy consumption by 11.2× compared to state-of-the-art methods.
Introduction
Deep Neural Networks (DNNs) are notoriously resource-intensive, complicating their deployment on edge devices like smartphones and IoT sensors. Quantization—representing weights and activations with fewer bits (e.g., moving from 32-bit floats to 8-bit integers)—is the standard solution. However, current methods face a trade-off: uniform quantization (like INT8) is hardware-friendly but often inaccurate for bell-curve weight distributions, while non-uniform methods are accurate but computationally expensive due to complex de-quantization loops. This paper proposes AFP to bridge this gap, offering a hardware-efficient, non-uniform format that works "out of the box."
The Adaptive Floating-Point (AFP) Framework
The core innovation is the AFP format, which allows for layer-by-layer configuration of bit-widths. Unlike standard floating-point formats with fixed segments, AFP enables the optimization of exponent bits ($n_{exp}$) and mantissa bits ($n_{man}$) to match the specific data distribution of each layer.
Key components of the framework include:
- Tunable Bias ($k$): A per-layer bias term shifts the representable range to cover specific weight distributions.
- Bayesian Optimization (BO) Search: The framework automatically identifies the optimal bit configuration for each layer, balancing accuracy (via KL-divergence minimization) against hardware costs.
- Simplified Arithmetic: AFP's power-of-two basis allows expensive multiplications to be replaced by simple bit-shifts and additions, reducing silicon area and power requirements.
Experimental Results
The researchers evaluated AFP using the ImageNet (ILSVRC 2012) dataset on ResNet-50 and MobileNet-v2. MobileNet-v2 is particularly challenging to quantize due to its compact architecture, yet AFP achieved a 71.59% accuracy with only ~5.7 bits (a minor 0.29% drop).
On ResNet-50, AFP achieved 76.09% Top-1 accuracy with an average of 4.8 bits, representing a negligible 0.04% drop from the full-precision baseline. Crucially, AFP reduced energy consumption by 11.2× for inference and achieved 9.3× power savings in Multiply-Accumulate (MAC) operations compared to standard floating-point implementations.
Discussion and Insights
AFP succeeds by addressing the "Gaussian Mismatch" inherent in uniform quantization. Because most neural network weights cluster near zero, uniform INT8 quantization wastes resolution on empty space at the distribution edges. AFP’s logarithmic nature naturally provides higher resolution where the data is most dense.
Furthermore, by allowing math to stay within the compressed format, AFP eliminates the "computational tax" of converting data back to FP32 during inference. While full realization of these gains requires specialized hardware (ASIC/FPGA) to handle custom bit-shifts, the framework provides a significant leap forward in post-training efficiency. Future work may explore extending these adaptive formats to higher-precision tasks like scientific computing or generative modeling.
Summary updated