OMPQ: Orthogonal Mixed Precision Quantization

Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Guannan Jiang, Wei Zhang, Rongrong Ji

Abstract

Mixed-precision quantization is essential for deploying deep neural networks on resource-constrained edge devices, yet finding the optimal bit-width for each layer remains a significant computational bottleneck. We present OMPQ (Orthogonal Mixed Precision Quantization), a highly efficient method that replaces iterative search algorithms with a geometric approach. By introducing the Orthogonality Metric (ORM) as a proxy for model accuracy, OMPQ transforms bit-allocation into a linear programming problem. Our method requires zero search iterations and as few as 32 images to achieve state-of-the-art results, reducing search time from hours to seconds.

Introduction

To run deep learning models on edge devices like smartphones or IoT sensors, quantization—reducing the precision of weights and activations from 32-bit floats to low-bit integers—is a standard practice. Mixed-precision quantization is particularly effective because it allows sensitive layers to maintain higher precision while aggressively compressing less critical ones.

However, the search space for bit configurations is exponentially large. Existing solutions typically rely on reinforcement learning, evolutionary algorithms, or second-order Hessian information. These methods are computationally expensive, often requiring hundreds of iterations, substantial training data, and significant GPU time. OMPQ addresses this "search bottleneck" by providing a deterministic, data-efficient alternative.

Methodology

OMPQ shifts the quantization problem from an iterative search to a geometric optimization. The core of the approach is the Orthogonality Hypothesis: a layer's mathematical orthogonality—the independence of its representation relative to others—is a strong indicator of its sensitivity to precision loss.

The OMPQ Pipeline

  1. Network Deconstruction: The neural network is viewed as a sequence of functions. The method analyzes the representational capability of these functions using a small batch of data.
  2. ORM Calculation: Calculating true orthogonality is mathematically complex. OMPQ uses Monte Carlo sampling and a Frobenius norm-based acceleration trick to approximate the Orthogonality Metric (ORM) in a single forward pass.
  3. Linear Programming (LP): Bit allocation is formulated as a constrained optimization problem. The objective is to maximize the total weighted orthogonality of the network while staying within a specific model size budget. This LP problem is solved almost instantaneously on a standard CPU.
The OMPQ pipeline: (a) Network deconstruction, (b) ORM matrix calculation, and (c) Linear programming for bit allocation.

Key Insights

  • Orthogonality as an Accuracy Proxy: The researchers identified a strong positive correlation between a model's total ORM and its Top-1 accuracy. This allows the ORM to serve as a reliable stand-in for accuracy during the configuration phase, eliminating the need for expensive validation cycles.
  • Efficiency and Data Sparsity: Unlike prior methods that may require the entire training set or over 1,000 images, OMPQ achieves peak performance using only 32 to 64 images.
  • Layer Sensitivity: Analysis using ORM revealed that $3 \times 3$ convolutional layers are generally more orthogonal than $1 \times 1$ layers. This explains why architectures like MobileNetV2, which rely heavily on $1 \times 1$ convolutions, are notoriously difficult to quantize and benefit most from OMPQ's precise bit allocation.

Experimental Results

We evaluated OMPQ on the ImageNet (ILSVRC12) dataset across ResNet-18, ResNet-50, and MobileNetV2 architectures.

Performance and Speed

  • Search Speed: For MobileNetV2, the bit-width search takes only 9 seconds on a CPU, whereas reinforcement learning methods can take hours or even days.
  • Accuracy: On ResNet-18, OMPQ achieved 72.08% Top-1 accuracy at a 6.7Mb model size, outperforming HAWQ-V3 (70.22%). On MobileNetV2, OMPQ reached 71.39% accuracy at 1.5Mb, significantly surpassing the BRECQ baseline of 70.28%.
Comparison of searching cost (iterations) and data requirements between OMPQ and existing methods like FracBits, HAWQ, and BRECQ.

As illustrated above, OMPQ reduces the "Searching Data" and "Searching Iterations" by orders of magnitude compared to established baselines, effectively achieving a "zero-iteration" search.

Conclusion and Future Work

OMPQ demonstrates that geometric properties of neural networks can be leveraged to bypass expensive search processes in model compression. While currently focused on CNNs, the principles of orthogonality could potentially be applied to Transformers and Large Language Models (LLMs). Future research will explore more complex optimization formulations, such as treating bit-allocation as a "multiple knapsack problem" to further refine the trade-offs between hardware constraints and model performance.

Summary updated

Abstract

To bridge the ever increasing gap between deep neural networks' complexity and hardware capability, network quantization has attracted more and more research attention. The latest trend of mixed precision quantization takes advantage of hardware's multiple bit-width arithmetic operations to unleash the full potential of network quantization. However, this also results in a difficult integer programming formulation, and forces most existing approaches to use an extremely time-consuming search process even with various relaxations. Instead of solving a problem of the original integer programming, we propose to optimize a proxy metric, the concept of network orthogonality, which is highly correlated with the loss of the integer programming but also easy to optimize with linear programming. This approach reduces the search time and required data amount by orders of magnitude, with little compromise on quantization accuracy. Specifically, we achieve 72.08% Top-1 accuracy on ResNet-18 with 6.7Mb, which does not require any searching iterations. Given the high efficiency and low data dependency of our algorithm, we used it for the post-training quantization, which achieve 71.27% Top-1 accuracy on MobileNetV2 with only 1.5Mb. Our code is available at https://github.com/MAC-AutoML/OMPQ.
BibTeX
@inproceedings{ma2023ompq,
  author = {Yuexiao Ma and Taisong Jin and Xiawu Zheng and Yan Wang and Huixia Li and Guannan Jiang and Wei Zhang and Rongrong Ji},
  title = {{OMPQ: Orthogonal Mixed Precision Quantization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i7.26084},
}
Figure