Searching for Low-Bit Weights in Quantized Neural Networks

Zhaohui Yang, Yunhe Wang, Kai Han, Chunjing Xu, Chao Xu, Dacheng Tao, Chang Xu

Introduction

Deep neural networks are notoriously memory-hungry and computationally expensive, making them difficult to deploy on mobile devices or specialized AI hardware. Quantization—representing weights with just a few bits (e.g., 1-bit or 2-bit) instead of 32-bit floats—is a powerful solution.

This paper introduces SLB (Searching for Low-Bit weights), a novel approach that treats neural network quantization as a search problem rather than a rounding problem. By representing each weight as a probability distribution over a discrete set of values and using a differentiable "continuous relaxation" strategy, the authors achieve state-of-the-art accuracy on low-bit models, significantly narrowing the performance gap between quantized and full-precision networks.

The Challenge of Quantization

There is a major mathematical hurdle in standard quantization: the functions used to "bin" weights into discrete values are non-differentiable (their gradients are zero almost everywhere). To train these networks, researchers usually rely on the Straight-Through Estimator (STE), which essentially "fakes" the gradient. While functional, STE provides inaccurate optimization directions. Because the weights are forced into hard discrete bins during the forward pass but updated using continuous gradients in the backward pass, the optimization process becomes unstable, leading to a significant drop in accuracy compared to full-precision models.

The authors propose a shift in perspective: if a 4-bit weight can only be one of 16 possible values, why not search for the best value among those 16?

Comparison of weight updates: standard discrete updates (left) versus continuous probability updates in SLB (right).

To achieve this, SLB employs several key mechanisms:

  1. Continuous Relaxation: Instead of a single weight value, the network maintains an auxiliary tensor that represents a probability distribution over a set of discrete candidate values (e.g., $\{-1, 0, 1\}$ for ternary weights).
  2. Softmax with Temperature ($\tau$): A Softmax function computes the expected value of a weight. During training, a "temperature" parameter controls the entropy of this distribution.
  3. Temperature Annealing: Training starts with a high temperature, making the weights a "fuzzy" blend of all possibilities. The temperature is gradually decreased toward zero. As $\tau \to 0$, the distribution becomes "sharp," eventually collapsing into a single discrete value (a one-hot vector).
  4. State Batch Normalization (SBN): Because the network uses "blended" continuous weights during training but "hard" discrete weights during inference, standard Batch Normalization statistics become inaccurate. SBN maintains two sets of statistics—one for training and one for inference—to ensure the model remains stable during this transition.

Key Insights and Ideas

  • Quantization as Search: By framing quantization as a search problem (similar to Neural Architecture Search), the authors bypass the need for inaccurate gradient estimators like STE. The entire training process becomes end-to-end differentiable.
  • The "Exp" Scheduler is King: The authors tested linear, cosine, and exponential temperature decay schedules. They found that an exponential scheduler allows the weights to converge to better local minima, leading to higher final accuracy.
Validation accuracy over epochs using different temperature decay schedulers.
  • Bridging the Training-Inference Divide: The introduction of State Batch Normalization (SBN) highlights that the mismatch between training (continuous) and testing (discrete) isn't just about the weights themselves, but also about how those weights interact with layer statistics.

Experimental Results

The authors evaluated SLB across major benchmarks, including CIFAR-10, ImageNet (ILSVRC2012), and Set5 (for Image Super-Resolution). SLB consistently outperformed existing state-of-the-art (SOTA) quantization methods like DoReFa-Net, DSQ, and LQ-Net.

  • ImageNet (ResNet18): In the extreme 1-bit (binary) weight/activation setting, SLB achieved 61.3% Top-1 accuracy. This is a massive jump over the original BNN (42.2%) and significantly higher than previous SOTA methods like BiRealNet (56.4%).
  • Near-Lossless 4-bit: At 4-bit weights and activations, SLB achieved 92.1% on CIFAR-10, nearly identical to the 32-bit full-precision baseline.
  • Super-Resolution: SLB recovered textures and details in upscaled images much more effectively than DoReFa-Net, performing very close to the full-precision VDSR model.

Limitations and Future Directions

While the final model is highly efficient, the training process requires auxiliary tensors to store probability distributions for every weight. This increases the memory footprint during the training phase, though it does not impact deployment. Additionally, while the paper focuses heavily on searching for weights, the activations are still quantized using more traditional methods. A future direction could involve applying a similar search-based strategy to the activation quantization thresholds. Finally, while the paper demonstrates theoretical efficiency (e.g., 32x compression for binary), translating this into actual speedups on specific hardware like FPGAs or mobile CPUs remains an area for practical implementation.

Summary updated

Abstract

Quantized neural networks with low-bit weights and activations are attractive for developing AI accelerators. However, the quantization functions used in most conventional quantization methods are non-differentiable, which increases the optimization difficulty of quantized networks. Compared with full-precision parameters (\emph{i.e.}, 32-bit floating numbers), low-bit values are selected from a much smaller set. For example, there are only 16 possibilities in 4-bit space. Thus, we present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately. In particular, each weight is represented as a probability distribution over the discrete value set. The probabilities are optimized during training and the values with the highest probability are selected to establish the desired quantized network. Experimental results on benchmarks demonstrate that the proposed method is able to produce quantized neural networks with higher performance over the state-of-the-arts on both image classification and super-resolution tasks.
BibTeX
@inproceedings{yang2020searching,
  author = {Zhaohui Yang and Yunhe Wang and Kai Han and Chunjing Xu and Chao Xu and Dacheng Tao and Chang Xu},
  title = {{Searching for Low-Bit Weights in Quantized Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=rA0N7Ed2ynxh},
}
Figure