Abstract: Low-bit quantization is challenging to maintain high performance with limited model capacity (e.g., 4-bit for both weights and activations). Naturally, the distribution of both weights and activations in deep neural network are Gaussian-like. Nevertheless, due to the limited bitwidth of low-bit model, uniform-like distributed weights and activations have been proved to be more friendly to quantization while preserving accuracy. Motivated by this, we propose Scale-Clip, a Distribution Reshaping technique that can reshape weights or activations into a uniform-like distribution in a dynamic manner. Furthermore, to increase the model capability for a low-bit model, a novel Group-based Quantization algorithm is proposed to split the filters into several groups. Different groups can learn different quantization parameters, which can be elegantly merged into batch normalization layer without extra computational cost in the inference stage. Finally, we integrate Scale-Clip technique with Group-based Quantization algorithm and propose the Group-based Distribution Reshaping Quantization (GDRQ) framework to further improve the quantization performance. Experiments on various networks (e.g. VGGNet and ResNet) and vision tasks (e.g. classification, detection, and segmentation) demonstrate that our framework achieves much better performance than state-of-the-art quantization methods. Specifically, the ResNet-50 model with 2-bit weights and 4-bit activations obtained by our framework achieves less than 1% accuracy drop on ImageNet classification task, which is a new state-of-the-art to our best knowledge.
TL;DR
Standard deep learning models have bell-shaped weight distributions that are notoriously difficult to squeeze into low-bit integers. This paper introduces GDRQ, a framework that reshapes these distributions into a uniform-like "flat" shape during training, allowing 2-bit weights and 4-bit activations to achieve near-lossless accuracy on ImageNet without any extra computational cost during inference.
What Problem Does This Paper Address?
To deploy deep neural networks on edge devices (like mobile phones or FPGAs), we often use quantization—converting 32-bit floating-point numbers into low-bit integers (e.g., 4-bit or 2-bit).
The challenge is that weights and activations in a typical network naturally follow Gaussian or Laplace distributions (bell curves), where most values cluster around zero. However, hardware-efficient linear quantization uses evenly spaced "bins." When you try to map a bell curve to these even bins, you either get massive errors at the tails (clipping) or lose all precision in the center where most of the data lives.
Previous methods focused on finding the "best" bins for a given distribution. This paper flips the script: Why not force the model to learn a distribution that fits the bins?
Methods
The authors propose the Group-based Distribution Reshaping Quantization (GDRQ) framework, which consists of two main components:
-
Scale-Clip (Distribution Reshaping): Instead of just clipping outliers to minimize error, the authors use a dynamic clipping threshold ($T$) during training. By forcing values into a specific range based on the mean of the absolute values ($T = k \cdot \text{mean}(|W|)$), the model is incentivized to spread its weights out. When $k \approx 2$, the distribution naturally flattens into a uniform-like shape, which is the "goldilocks" zone for linear quantization.
-
Group-based Quantization: To increase the model's expressive power, they split convolutional filters into several groups. Each group learns its own optimal quantization parameters (clipping values).
- The "Free Lunch": Crucially, these different group-wise parameters can be mathematically merged into the following Batch Normalization (BN) layer. This means the model gets the benefit of high-granularity quantization during training, but the final deployed model runs just as fast as a standard quantized network.
Datasets
The authors validated their approach across several standard benchmarks to prove it wasn't a "one-hit wonder":
- Image Classification: ImageNet (ILSVRC-12) and CIFAR-100.
- Object Detection: PASCAL VOC (using Faster R-CNN).
- Semantic Segmentation: Cityscapes (using PSPNet).
Key Insights and Ideas
- Uniform is Better: The paper theoretically and experimentally proves that a uniform distribution results in significantly lower quantization loss compared to Gaussian or Laplace distributions when using linear quantizers.
- Reshaping vs. Fitting: Most quantization research treats the pre-trained distribution as "holy" and tries to fit bins to it. This paper shows that the distribution itself is flexible; if you penalize the model for being "bell-shaped" during fine-tuning, it will adapt without losing accuracy.
- Stability Matters: While smaller groups (more parameters) generally help, the authors found that if a group is too small (e.g., a group size of 1), the statistical measures become unstable, which can hurt performance. There is a sweet spot for group size that balances capacity and stability.
Key Results
The results are among the best for 2-bit/4-bit regimes at the time of publication:
- ResNet-50 (ImageNet): Achieved 73.9% Top-1 accuracy with 2-bit weights and 4-bit activations. This is less than a 1% drop from the floating-point baseline (74.8%), outperforming other popular methods like SYQ and FGQ.
- VGG-16 (ImageNet): With 4-bit weights and activations, the model reached 72.5%, nearly identical to the floating-point version (72.6%).
- Detection & Segmentation: On PASCAL VOC, the 2-bit/4-bit model saw only a 0.7% mAP drop. On Cityscapes, the mIoU drop was only 0.9%, proving the method generalizes well beyond simple classification.
Limitations and Future Work
- Hyper-parameter Sensitivity: The "k" factor in Scale-Clip and the group size "gs" require some tuning. The authors suggest $k=2$ is generally best for weights, but activations require a larger $k$ and a more complex update strategy because their distributions are more volatile during training.
- Extreme Low-Bit: While 2-bit weights perform excellently, the paper notes that moving to 1-bit (binary) weights still results in a more significant accuracy gap, especially when the group size is very small.
Summary updated