Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization

Weihan Chen, Peisong Wang, Jian Cheng

Deep Neural Networks (DNNs) are notoriously memory-intensive and computationally expensive, posing significant challenges for deployment on edge devices. While quantization—reducing precision from 32-bit floating point to lower bit-widths—is a standard solution, traditional uniform quantization often yields suboptimal results. This paper introduces a principled and highly efficient framework for mixed-precision quantization, where different layers are assigned varying bit-widths to optimally balance model size and accuracy.

The Mixed-Precision Challenge

The primary hurdle in mixed-precision quantization is the exponential search space. For a network with $N$ layers and $M$ possible bit-widths, there are $M^N$ potential combinations. Previous approaches relying on Reinforcement Learning (RL) or manual heuristics are often prohibitively slow, sometimes requiring thousands of GPU hours to find a viable configuration.

Methodology: From Taylor Expansion to Knapsack Problems

The authors reformulate the search for bit-widths as a discrete constrained optimization problem. Their approach follows three key mathematical steps:

  1. Second-Order Taylor Approximation: Evaluating the true loss for every bit-combination is impossible. Instead, the authors approximate loss perturbation using the Hessian matrix (the second-order derivative of the loss). To manage computational costs, they utilize a Jacobian-based approximation that requires only one row of the Jacobian per sample.
  2. MCKP Reformulation: By assuming layer independence (a block-diagonal Hessian), the problem becomes mathematically equivalent to the Multiple-Choice Knapsack Problem (MCKP). Each layer acts as a "class," and each bit-width is an "item" with a specific memory cost ("weight") and a corresponding reduction in loss perturbation ("profit").
  3. Efficient Greedy Search: The resulting MCKP is solved using a greedy algorithm that iteratively selects the layer-bit combination providing the highest "bang for the buck" until the memory or computation budget is reached.
Figure 2: Convergence of loss perturbation estimates across different architectures and bit-widths. The sensitivity values stabilize rapidly, requiring only a small fraction of the dataset.

Key Insights: Sensitivity and Convergence

A core finding of the research is that the Hessian matrix effectively represents a layer's "sensitivity." High curvature in the loss surface indicates that quantization noise in that layer will significantly degrade performance. Crucially, the authors discovered that these sensitivity estimates converge rapidly. As shown in Figure 2, loss perturbation values stabilize after seeing only about 1024 images, allowing the search phase to be completed in minutes rather than days.

Experimental Results

The framework was evaluated on the ImageNet dataset using ResNet-18, ResNet-50, and MobileNet-V2.

  • Extreme Efficiency: For ResNet-50, the entire search process takes less than two minutes on a single RTX 2080Ti—orders of magnitude faster than RL-based methods like HAQ.
  • Superior Accuracy: The second-order approximation consistently outperforms uniform bit-width assignments and first-order (gradient-only) methods. For example, on ResNet-18, the method achieved a 10.66x weight compression with only a 0.26% drop in Top-1 accuracy.
Figure 3: Top-1 Accuracy vs. Compression Ratio across ResNet-18, ResNet-50, and MobileNet-V2. The proposed second-order Taylor approximation (purple line) significantly outperforms uniform and first-order baselines.

As illustrated in Figure 3, the proposed method maintains high accuracy even at high compression ratios where uniform quantization (orange dashed line) fails. This is particularly evident in MobileNet-V2, a lightweight architecture that is notoriously difficult to quantize.

Conclusion

By treating mixed-precision quantization as a constrained optimization problem and leveraging second-order sensitivity analysis, this framework provides a theoretically grounded and practically fast solution for model compression. The ability to find optimal bit-width assignments in minutes makes this approach highly scalable for modern, complex neural architectures.

Summary updated

Abstract

Quantization is a widely used technique to compress and accelerate deep neural networks. However, conventional quantization methods use the same bit-width for all (or most of) the layers, which often suffer significant accuracy degradation in the ultra-low precision regime and ignore the fact that emergent hardware accelerators begin to support mixed-precision computation. Consequently, we present a novel and principled framework to solve the mixed-precision quantization problem in this paper. Briefly speaking, we first formulate the mixed-precision quantization as a discrete constrained optimization problem. Then, to make the optimization tractable, we approximate the objective function with second-order Taylor expansion and propose an efficient approach to compute its Hessian matrix. Finally, based on the above simplification, we show that the original problem can be reformulated as a Multiple-Choice Knapsack Problem (MCKP) and propose a greedy search algorithm to solve it efficiently. Compared with existing mixed-precision quantization works, our method is derived in a principled way and much more computationally efficient. Moreover, extensive experiments conducted on the ImageNet dataset and various kinds of network architectures also demonstrate its superiority over existing uniform and mixed-precision quantization approaches.
BibTeX
@inproceedings{chen2021towards,
  author = {Weihan Chen and Peisong Wang and Jian Cheng},
  title = {{Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00530},
}
Figure