TL;DR
OMPQ (Orthogonal Mixed Precision Quantization) introduces a novel proxy metric based on network function orthogonality to solve the mixed-precision quantization problem. By framing bit allocation as a simple linear programming problem rather than a grueling combinatorial search, OMPQ eliminates the need for searching iterations entirely. This reduces search time to mere seconds and data requirements to a handful of images, all while achieving state-of-the-art quantization accuracy.
What Problem Does This Paper Address?
To deploy deep neural networks on resource-constrained edge hardware, developers use network quantization—converting 32-bit floating-point weights and activations into lower-bit integers (like 4-bit or 8-bit). Mixed-precision quantization takes this a step further by assigning different bit-widths to different layers, maximizing compression without tanking accuracy.
However, finding the optimal bit configuration for each layer is a nightmare. It is a non-differentiable, highly non-convex integer programming problem. Existing state-of-the-art methods rely on heavy, time-consuming search mechanisms like reinforcement learning, evolutionary algorithms, or calculating Hessian eigenvalues. These methods often require hundreds of iterations and thousands to millions of images, creating a massive computational bottleneck.
Methods
Instead of relaxing constraints or running expensive black-box optimization loops, the authors propose a proxy metric called the ORthogonality Metric (ORM).
The core workflow consists of three main steps:
- Network Deconstruction: The network is deconstructed into a set of cumulative functions, where each function represents the transformation from the input up to that specific layer.
- ORM Calculation: Using a tiny batch of calibration data, the authors compute the orthogonality between these layer functions via Monte Carlo sampling. To make this mathematically tractable across different layer dimensions, they normalize the metric between 0 and 1 using the Cauchy-Schwarz inequality.
- Linear Programming: The authors prove that a layer's orthogonality is highly correlated with its sensitivity to quantization. They convert the ORM values into layer importance factors ($\theta$) and set up a linear programming problem to maximize total model orthogonality under a strict model size constraint.
To make this practical, the authors also introduced a Calculation Acceleration technique. Computing ORM directly on large feature maps has a high time complexity. By deriving an equivalent form using matrix flattening (vec(YY^T)), they shifted the complexity to scale with the number of samples rather than the feature dimensions, speeding up the calculation by up to 54$\times$ when the sample size is small.
Datasets
The method was evaluated on the standard ImageNet dataset (1.2M training images, 50,000 validation images). Crucially, for the actual mixed-precision bit searching phase, OMPQ only required a tiny fraction of data:
- 64 random samples for ResNet-18 and ResNet-50.
- 32 random samples for MobileNetV2.
Key Insights and Ideas
- Orthogonality Equals Accuracy: The authors discovered a strong positive correlation between a model's total orthogonality and its final quantized accuracy. Maximizing orthogonality implicitly minimizes the representation error of the quantized network.
- Layer Sensitivity Insight: Layers with stronger orthogonality against all other layers are more independent and contain unique representational information. Therefore, the linear programming model naturally assigns higher bit-widths to these highly orthogonal layers.
- Why MobileNetV2 is Hard to Quantize: Layer-wise analysis revealed that $3\times3$ depthwise convolutional layers are significantly more orthogonal than $1\times1$ pointwise convolutional layers. This explains why architectures heavily reliant on $1\times1$ convolutions suffer severe accuracy degradation under uniform low-bit quantization—they require a more nuanced, mixed-precision allocation.
- Depth-Dependent Bit Allocation: Across both ResNet and MobileNet architectures, the optimal bit-width configuration generally decreases as the depth of the layer increases.
Key Results
OMPQ slashes the resource requirements for mixed-precision configuration down to near-zero while matching or exceeding the accuracy of methods that take hours to run.
- Zero Search Iterations: Unlike HAWQ (1050 iterations) or BRECQ (100 iterations), OMPQ requires 0 searching iterations. The linear programming problem solves in less than 10 seconds on a single CPU.
- Quantization-Aware Training (QAT): On ResNet-18, OMPQ achieved 72.08% Top-1 accuracy with a model size of just 6.7Mb, outperforming HAWQ-V3 by 1.86% at a similar size. On ResNet-50, it achieved 76.28% Top-1 accuracy at 18.7Mb.
- Post-Training Quantization (PTQ): When integrated with the BRECQ reconstruction block for PTQ on MobileNetV2, OMPQ achieved 71.27% Top-1 accuracy with only 1.5Mb of parameters, outperforming FracBits by 1.37% despite using a smaller model size.
Limitations and Future Work
While OMPQ provides an incredibly elegant and fast solution for bit-width allocation, it relies on a hyperparameter ($\beta$) to control the variance of the importance factors. If $\beta$ is set too close to zero, the variance drops, which can cause the linear programming solver to default to overly aggressive, sub-optimal bit configurations.
For future work, the authors plan to expand this framework by modeling the mixed-precision quantization problem as a multiple-knapsack problem combined with their network orthogonality metric to handle even more complex hardware constraints.
Summary updated