OMPQ: Orthogonal Mixed Precision Quantization

Yuexiao Ma, Taisong Jin, Xiawu Zheng, Yan Wang, Huixia Li, Yongjian Wu, Guannan Jiang, Wei Zhang, Rongrong Ji

TL;DR

OMPQ (Orthogonal Mixed Precision Quantization) introduces a novel proxy metric based on network function orthogonality to solve the mixed-precision quantization problem. By framing bit allocation as a simple linear programming problem rather than a grueling combinatorial search, OMPQ eliminates the need for searching iterations entirely. This reduces search time to mere seconds and data requirements to a handful of images, all while achieving state-of-the-art quantization accuracy.

What Problem Does This Paper Address?

To deploy deep neural networks on resource-constrained edge hardware, developers use network quantization—converting 32-bit floating-point weights and activations into lower-bit integers (like 4-bit or 8-bit). Mixed-precision quantization takes this a step further by assigning different bit-widths to different layers, maximizing compression without tanking accuracy.

However, finding the optimal bit configuration for each layer is a nightmare. It is a non-differentiable, highly non-convex integer programming problem. Existing state-of-the-art methods rely on heavy, time-consuming search mechanisms like reinforcement learning, evolutionary algorithms, or calculating Hessian eigenvalues. These methods often require hundreds of iterations and thousands to millions of images, creating a massive computational bottleneck.

Methods

Instead of relaxing constraints or running expensive black-box optimization loops, the authors propose a proxy metric called the ORthogonality Metric (ORM).

The core workflow consists of three main steps:

  1. Network Deconstruction: The network is deconstructed into a set of cumulative functions, where each function represents the transformation from the input up to that specific layer.
  2. ORM Calculation: Using a tiny batch of calibration data, the authors compute the orthogonality between these layer functions via Monte Carlo sampling. To make this mathematically tractable across different layer dimensions, they normalize the metric between 0 and 1 using the Cauchy-Schwarz inequality.
  3. Linear Programming: The authors prove that a layer's orthogonality is highly correlated with its sensitivity to quantization. They convert the ORM values into layer importance factors ($\theta$) and set up a linear programming problem to maximize total model orthogonality under a strict model size constraint.
Figure 2: Overview. Left: Deconstruct the model into a set of functions F. Middle: ORM symmetric matrix calculated from F. Right: Linear programming problem constructed by the importance factor θ to derive optimal bit configuration.

To make this practical, the authors also introduced a Calculation Acceleration technique. Computing ORM directly on large feature maps has a high time complexity. By deriving an equivalent form using matrix flattening (vec(YY^T)), they shifted the complexity to scale with the number of samples rather than the feature dimensions, speeding up the calculation by up to 54$\times$ when the sample size is small.

Datasets

The method was evaluated on the standard ImageNet dataset (1.2M training images, 50,000 validation images). Crucially, for the actual mixed-precision bit searching phase, OMPQ only required a tiny fraction of data:

  • 64 random samples for ResNet-18 and ResNet-50.
  • 32 random samples for MobileNetV2.

Key Insights and Ideas

  • Orthogonality Equals Accuracy: The authors discovered a strong positive correlation between a model's total orthogonality and its final quantized accuracy. Maximizing orthogonality implicitly minimizes the representation error of the quantized network.
  • Layer Sensitivity Insight: Layers with stronger orthogonality against all other layers are more independent and contain unique representational information. Therefore, the linear programming model naturally assigns higher bit-widths to these highly orthogonal layers.
  • Why MobileNetV2 is Hard to Quantize: Layer-wise analysis revealed that $3\times3$ depthwise convolutional layers are significantly more orthogonal than $1\times1$ pointwise convolutional layers. This explains why architectures heavily reliant on $1\times1$ convolutions suffer severe accuracy degradation under uniform low-bit quantization—they require a more nuanced, mixed-precision allocation.
  • Depth-Dependent Bit Allocation: Across both ResNet and MobileNet architectures, the optimal bit-width configuration generally decreases as the depth of the layer increases.

Key Results

OMPQ slashes the resource requirements for mixed-precision configuration down to near-zero while matching or exceeding the accuracy of methods that take hours to run.

Figure 1: Comparison of the resources used to obtain the optimal bit configuration between our algorithm and other mixed precision algorithms (FracBits (Yang and Jin 2021), HAWQ (Dong et al. 2020), BRECQ (Li et al. 2021)) on ResNet-18. “Searching Data” is the number of input images.
  • Zero Search Iterations: Unlike HAWQ (1050 iterations) or BRECQ (100 iterations), OMPQ requires 0 searching iterations. The linear programming problem solves in less than 10 seconds on a single CPU.
  • Quantization-Aware Training (QAT): On ResNet-18, OMPQ achieved 72.08% Top-1 accuracy with a model size of just 6.7Mb, outperforming HAWQ-V3 by 1.86% at a similar size. On ResNet-50, it achieved 76.28% Top-1 accuracy at 18.7Mb.
  • Post-Training Quantization (PTQ): When integrated with the BRECQ reconstruction block for PTQ on MobileNetV2, OMPQ achieved 71.27% Top-1 accuracy with only 1.5Mb of parameters, outperforming FracBits by 1.37% despite using a smaller model size.

Limitations and Future Work

While OMPQ provides an incredibly elegant and fast solution for bit-width allocation, it relies on a hyperparameter ($\beta$) to control the variance of the importance factors. If $\beta$ is set too close to zero, the variance drops, which can cause the linear programming solver to default to overly aggressive, sub-optimal bit configurations.

For future work, the authors plan to expand this framework by modeling the mixed-precision quantization problem as a multiple-knapsack problem combined with their network orthogonality metric to handle even more complex hardware constraints.

Summary updated

Abstract

To bridge the ever increasing gap between deep neural networks' complexity and hardware capability, network quantization has attracted more and more research attention. The latest trend of mixed precision quantization takes advantage of hardware's multiple bit-width arithmetic operations to unleash the full potential of network quantization. However, this also results in a difficult integer programming formulation, and forces most existing approaches to use an extremely time-consuming search process even with various relaxations. Instead of solving a problem of the original integer programming, we propose to optimize a proxy metric, the concept of network orthogonality, which is highly correlated with the loss of the integer programming but also easy to optimize with linear programming. This approach reduces the search time and required data amount by orders of magnitude, with little compromise on quantization accuracy. Specifically, we achieve 72.08% Top-1 accuracy on ResNet-18 with 6.7Mb, which does not require any searching iterations. Given the high efficiency and low data dependency of our algorithm, we used it for the post-training quantization, which achieve 71.27% Top-1 accuracy on MobileNetV2 with only 1.5Mb. Our code is available at https://github.com/MAC-AutoML/OMPQ.
BibTeX
@inproceedings{ma2023ompq,
  author = {Yuexiao Ma and Taisong Jin and Xiawu Zheng and Yan Wang and Huixia Li and Yongjian Wu and Guannan Jiang and Wei Zhang and Rongrong Ji},
  title = {{OMPQ: Orthogonal Mixed Precision Quantization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i7.26084},
}
Figure